We’re excited to announce the Normal Availability of Hive Metastore (HMS) and AWS Glue Federation in Unity Catalog! This new functionality allows Unity Catalog to seamlessly entry and govern tables saved in Hive Metastores—whether or not self-hosted or hosted by Databricks—in addition to AWS Glue. It represents a key milestone in our Lakehouse Federation imaginative and prescient, which brings exterior information sources together with databases, information warehouses and catalogs, collectively underneath a unified governance framework with Unity Catalog. Now you can effortlessly uncover, question, and handle all of your information from a single, centralized platform, whatever the format and site. This not solely fosters open entry and collaboration throughout your group but in addition extends information intelligence into each information supply.
On this weblog, we’ll discover the advantages of HMS and AWS Glue Federation, clarify the way it works, and supply steerage on getting began.
Why Hive Metastore and AWS Glue Federation?
HMS has been an early customary for cataloging information to be used in huge information techniques, and whereas it offers foundational functionalities, they aren’t ideally fitted to fashionable information and AI workloads that demand complete governance, together with fine-grained entry controls on rows and columns, lineage, monitoring and auditing throughout all information and AI belongings in a single place.
Unity Catalog addresses these shortcomings by offering the trade’s solely unified, open governance answer for managing all information and AI belongings. It allows organizations to create an enterprise catalog that curates information, tables, ML fashions, AI instruments, notebooks, and metrics, all ruled with fine-grained entry controls, lineage, monitoring, auditing, and cross-platform sharing in a single answer. Over 10,000+ enterprises at the moment are leveraging Unity Catalog to control their information property.
HMS and AWS Glue Federation present important advantages for organizations with HMS deeply embedded of their information structure. For these with long-standing HMS or AWS Glue deployments, this functionality presents a seamless path to leverage Unity Catalog’s superior options over information saved within the HMS or Glue metastore. It ensures operational continuity by enabling organizations to maintain legacy workflows whereas progressively upgrading current information and workspaces to Unity Catalog.
Key advantages embody:
- Seamless integration: Join your current HMS and AWS Glue catalogs on to Unity Catalog with out requiring guide metadata migration.
- Simplified information discovery: Entry and discover information from HMS and AWS Glue by way of a unified interface, alongside all of your information and AI belongings in Unity Catalog.
- Complete governance: Leverage Unity Catalog’s fine-grained entry controls, tagging, classification, lineage, and audit capabilities on prime of the information saved in HMS and AWS Glue
Using HMS Federation in Unity Catalog enabled an incremental migration that minimized disruption to our prospects. Our partnership with Databricks was essential in evaluating short-term wants and long-term targets. Leveraging Unity Catalog’s capabilities early, we created a scalable and environment friendly information platform with improved governance and entry management as a part of our modernization journey
— Praveena Edward, Lead Information Engineer, Nationwide
We have now years’ price of datasets which can be cataloged in an exterior Hive Metastore. HMS Federation permits us to instantly profit from Unity Catalog solely options like sturdy entry management and self-serve AI tooling by way of Genie Areas, with out the overhead of migrating all of those tables into Unity Catalog
— James Davidheiser, Technical Lead, Information Infrastructure, Asana
The way it works
Unity Catalog now contains federation connectors for Hive Metastore (HMS) and AWS Glue, serving as a translation layer between Unity Catalog and your exterior metastores. These connectors allow you to mount whole HMS catalogs (each inside and exterior) or AWS Glue as international catalogs inside Unity Catalog, making them seem as native objects. You may outline fine-grained entry controls, view lineage, carry out audits, and question HMS or AWS Glue managed tables utilizing the Databricks engine. The federation helps each studying and writing to tables in inside HMS inside Databricks workspaces whereas providing read-only entry for tables in exterior HMS and AWS Glue
With this functionality, you possibly can learn all tables in HMS and AWS Glue—Parquet, Delta, and Iceberg (coming quickly in Public Preview)—enabling you to entry and govern all of your tables seamlessly.
Try the video tutorial under to discover AWS Glue and HMS Federation in motion.
What’s new with the GA launch?
The GA launch comes with the next further options:
- Help for Hive Metastore v3.1
- Help for Hive Metastores hosted in a SQL Server or Postgres databases (mySQL help already offered)
- Entry to DBFS-root backed tables behind a storage firewall (Azure)
- Help for row and column degree masks on Hive tables
We’ve additionally added Public Preview help for studying Delta shallow clones outlined within the Hive metastore, in addition to Non-public Preview help for studying DBFS-mount backed tables throughout workspaces (please contact your Databricks account consultant to enroll!).
Get began
By embracing Unity Catalog because the cornerstone of your Lakehouse structure, you possibly can unlock the facility of a unified and open governance implementation that spans your whole information and AI property.