Deprecation of Lake Formation’s Ruled Tables Function

After cautious consideration, we’ve made the choice to finish help for Ruled Tables, efficient December 31, 2024, to give attention to open supply transactional desk codecs resembling Apache Iceberg, Apache Hudi, and Linux Basis Delta Lake. This choice stems from buyer choice for these open supply options, which provide ACID-compliant transactions, compaction, time journey, and different options beforehand supplied by Ruled Tables. Amazon Net Companies (AWS) has enhanced our help for these codecs throughout varied analytics providers, together with Amazon Athena, Amazon EMR (Elastic MapReduce), AWS Glue, and Amazon Redshift, with options that embody automated compaction help for Apache Iceberg, retention and snapshot expiration and orphan file deletion for Apache Iceberg, enhancements to AWS Glue Knowledge Catalog CreateTable API to create Apache Iceberg tables, and AWS Glue Crawlers schema detection help throughout Apache Iceberg, Apache Hudi, and Linux Basis Delta Lake. Prospects can now use these open supply codecs to realize ACID-compliant transactions with Amazon Easy Storage Service (Amazon S3) backed knowledge, benefiting from their wealthy options and large compatibility.

After December 31, 2024, prospects will now not have the ability to create Ruled Tables transactions (lakeformation:StartTransaction), write to Ruled Tables (lakeformation:UpdateTableObjects), or question your Ruled Tables utilizing Amazon Athena. Prospects will nonetheless have the ability to entry their desk state info by calling the lakeformation:GetTableObjects and transaction info by calling lakeformation:ListTransactions till February 17, 2025. After February 17, 2025, all Ruled Desk APIs will begin to fail. Ruled Tables metadata will live on inside the AWS Glue Knowledge Catalog, and the Ruled Tables knowledge will stay in your S3 buckets. No different desk varieties might be affected by this alteration, together with Hive (Apache Parquet, CSV, ORC, and so forth), Iceberg, Hudi, and Delta Lake tables.

Migrating your Ruled Tables

Prospects can migrate their tables from Ruled Tables to one of many open supply codecs by copying their ruled desk knowledge on to Apache Iceberg utilizing Amazon Athena. Emigrate knowledge to Iceberg, you should utilize the Amazon Athena CREATE TABLE AS (CTAS) assertion, as proven within the following code instance.

CREATE TABLE my_iceberg_table WITH ( 
    table_type="ICEBERG",
    is_external = false,
    location = 's3://mybucket/myicebergdata/'
) AS 
SELECT * FROM my_governed_table

You possibly can specify extra table-level properties, that are listed within the Amazon Athena Consumer Information. Should you specify partitions or buckets as a part of the Apache Iceberg desk definition, then you could run into the 100 partition per bucket limitation. On this case, check with Use CTAS and INSERT INTO to work across the 100 partition restrict.

Should you require any help migrating your tables, or have any questions, attain out to us at [email protected].

In regards to the writer

Mert Hocanin is a Principal Massive Knowledge Architect with AWS Lake Formation.