Every little thing begins with good information, so ingestion is usually your first step to unlocking insights. Nonetheless, ingestion presents challenges, like ramping up on the complexities of every information supply, protecting tabs on these sources as they alter, and governing all of this alongside the way in which.
Lakeflow Join makes environment friendly information ingestion straightforward, with a point-and-click UI, a easy API, and deep integrations with the Knowledge Intelligence Platform. Final 12 months, greater than 2,000 prospects used Lakeflow Connect with unlock worth from their information.
On this weblog, we’ll overview the fundamentals of Lakeflow Join and recap the newest bulletins from the 2025 Knowledge + AI Summit.
Ingest all of your information in a single place with Lakeflow Join
Lakeflow Join provides easy ingestion connectors for functions, databases, cloud storage, message buses, and extra. Underneath the hood, ingestion is environment friendly, with incremental updates and optimized API utilization. As your managed pipelines run, we handle schema evolution, seamless third-party API upgrades, and complete observability with built-in alerts.
Knowledge + AI Summit 2025 Bulletins
At this 12 months’s Knowledge + AI Summit, Databricks introduced the Normal Availability of Lakeflow, the unified method to information engineering throughout ingestion, transformation, and orchestration. As a part of this, Lakeflow Join introduced Zerobus, a direct write API that simplifies ingestion for IoT, clickstream, telemetry and different related use instances. We additionally expanded the breadth of supported information sources with extra built-in connectors throughout enterprise functions, file sources, databases, and information warehouses, in addition to information from cloud object storage.
Zerobus: a brand new technique to push occasion information on to your lakehouse
We made an thrilling announcement introducing Zerobus, a brand new progressive method for pushing occasion information on to your lakehouse by bringing you nearer to the info supply. Eliminating information hops and decreasing operational burden allows Zerobus to supply high-throughput direct writes with low latency, delivering close to real-time efficiency at scale.
Beforehand, some organizations used message buses like Kafka as transport layers to the Lakehouse. Kafka provides a sturdy, low-latency method for information producers to ship information, and it’s a preferred alternative when writing to a number of sinks. Nonetheless, it additionally provides further complexity and prices, in addition to the burden of managing one other information copy—so it’s inefficient when your sole vacation spot is the Lakehouse. Zerobus offers a easy answer for these instances.
Joby Aviation is already utilizing Zerobus to instantly push telemetry information into Databricks.
Joby is ready to use our manufacturing brokers with Zerobus to push gigabytes a minute of telemetry information on to our lakehouse, accelerating the time to insights — all with Databricks Lakeflow and the Knowledge Intelligence Platform.”
— Dominik Müller, Manufacturing unit Methods Lead, Joby Aviation, Inc.
As a part of Lakeflow Join, Zerobus can be unified with the Databricks Platform, so you’ll be able to leverage broader analytics and AI capabilities instantly. Zerobus is at the moment in Non-public Preview; attain out to your account group for early entry.
🎥 Watch and be taught extra about Zerobus: Breakout session on the Knowledge + AI Summit, that includes Joby Aviation, “Lakeflow Join: eliminating hops in your streaming structure”
Lakeflow Join expands ingestion capabilities and information sources
New absolutely managed connectors are persevering with to roll out throughout numerous launch states (see full listing under), together with Google Analytics and ServiceNow, in addition to SQL Server – the primary database connector, all at the moment in Public Preview with Normal Availability coming quickly.
We’ve additionally continued innovating for purchasers who need extra customization choices and use our current ingestion answer, Auto Loader. It incrementally and effectively processes new information recordsdata as they arrive in cloud storage. We’ve launched some main value and efficiency enhancements for Auto Loader, together with 3X sooner listing listings and computerized cleanup with “CleanSource,” each now typically accessible, together with smarter and cheaper file discovery utilizing file occasions. We additionally introduced native help for ingesting Excel recordsdata and ingesting information from SFTP servers, each in Non-public Preview, accessible by request for early entry.
Supported information sources:
- Functions: Salesforce, Workday, ServiceNow, Google Analytics, Microsoft Dynamics 365, Oracle NetSuite
- File sources: S3, ADLS, GCS, SFTP, SharePoint
- Databases: SQL Server, Oracle Database, MySQL, PostgreSQL
- Knowledge warehouses: Snowflake, Amazon Redshift, Google BigQuery
Throughout the expanded connector providing, we’re introducing query-based connectors that simplify information ingestion. These new connectors will let you pull information instantly out of your supply methods with out database modifications and work with learn replicas the place change information seize (CDC) logs aren’t accessible. That is at the moment in Non-public Preview; attain out to your account group for early entry.
🎥 Watch and be taught extra about Lakeflow Join: Breakout session on the Knowledge + AI Summit, “Getting Began with Lakeflow Join”
🎥 Watch and be taught extra about ingesting from enterprise SaaS functions: Breakout session on the Knowledge + AI Summit that includes Databricks buyer Porsche Holding, “Lakeflow Join: Seamless Knowledge Ingestion From Enterprise Apps”
🎥 Watch and be taught extra about database connectors: Breakout session on the Knowledge + AI Summit, “Lakeflow Join: Straightforward, Environment friendly Ingestion From Databases”
Lakeflow Join in Jobs, now typically accessible
We’re persevering with to develop capabilities to make it simpler so that you can use our ingestion connectors whereas constructing information pipelines, as a part of Lakeflow’s unified information engineering expertise. Databricks not too long ago introduced Lakeflow Join in Jobs, which lets you create ingestion pipelines inside Lakeflow Jobs. So, when you’ve got jobs as the middle of your ETL course of, this seamless integration offers a extra intuitive and unified expertise for managing ingestion.
Clients can outline and handle their end-to-end workloads—from ingestion to transformation—multi functional place. Lakeflow Join in Jobs is now typically accessible.
🎥 Watch and be taught extra about Lakeflow Jobs: Breakout session on the Knowledge + AI Summit “Orchestration with Lakeflow Jobs”
Lakeflow Join: extra to come back in 2025 and past
Databricks understands the wants of knowledge engineers and organizations who drive innovation with their information utilizing analytics and AI instruments. To that finish, Lakeflow Join has continued to construct out sturdy, environment friendly ingestion capabilities with absolutely managed connectors to extra customizable options and APIs.
We’re simply getting began with Lakeflow Join. Keep tuned for extra bulletins later this 12 months, or contact your Databricks account group to hitch a preview for early entry.
To strive Lakeflow Join, you’ll be able to overview the documentation, or try the Demo Heart.