SAN FRANCISCO — Dremio, the Agentic Lakehouse firm, immediately highlighted its management throughout the Apache Iceberg ecosystem, together with V3 help now accessible in Dremio Cloud, the election of Dremio engineer JB Onofre to the Apache Software program Basis board, and continued momentum behind Apache Polaris. A longstanding advocate for open-source collaboration and the elimination of vendor lock-in, Dremio has made foundational contributions to tasks together with Apache Arrow (co-creator and core contributor), Apache Iceberg (contributor and main educator), and Apache Polaris (co-creator). Reinforcing this dedication, JB Onofre, who shepherded Polaris by incubation, has been elected to the Apache Software program Basis board.
Iceberg V3 is designed to help extra various and sophisticated knowledge varieties, supply larger management over schema evolution, and ship efficiency enhancements for large-scale, high-concurrency environments. Dremio’s V3 integration advances dealing with of semi-structured knowledge, row-level adjustments, and schema evolution, with full help in Dremio Cloud, together with the VARIANT knowledge sort for JSON, deletion vectors for quicker CDC (change knowledge seize), and improved schema evolution.
“The Iceberg lakehouse has change into the default structure for AI and analytics,” mentioned Rahim Bhojani, CTO of Dremio. “Most platforms added Iceberg as a characteristic, however Dremio was constructed on it from the bottom up. Capabilities like Autonomous Reflections, Iceberg Clustering, and now V3 compound on one another, delivering an Iceberg platform that’s each the quickest and the simplest to handle.”
Dremio continues to set the usual for Apache Iceberg with:
- Apache Iceberg V3 Assist: Dremio delivers full learn and write help for the newest Iceberg specification. Deletion vectors speed up row-level operations for CDC and streaming workloads. The VARIANT sort eliminates the schema-on-write bottleneck for semi-structured knowledge. Row-level lineage supplies built-in creation and replace monitoring for regulated industries with no extra tooling required.
- Arrow-Based mostly SQL Engine for Iceberg: Dremio’s question engine was constructed natively on Apache Arrow, the open columnar normal Dremio co-created, making it uniquely suited to Iceberg workloads. It processes Iceberg and Parquet knowledge in vectorized batches with out conversion to a proprietary format, delivering quick, scalable analytics with no lock-in.
- Autonomous Reflections: Dremio eliminates the administration overhead of working an Iceberg lakehouse. Autonomous Reflections observe question patterns and robotically creates, refreshes, and retires materializations, accelerating queries from seconds to sub-second with no code adjustments or handbook tuning. Reflections’ incremental refresh retains knowledge contemporary at low useful resource value.
- Iceberg Clustering: makes use of Z-order to co-locate knowledge throughout a number of columns concurrently. ith two-level pruning that skips knowledge at each the manifest and row-group degree, it minimizes I/O by working repeatedly on petabyte-scale tables with out full-table rewrites. Computerized desk upkeep: compaction, snapshot expiration, and orphan file cleanup run on policy-based schedules with no handbook intervention, retaining tables performant and storage prices in verify. Allows engineers to concentrate on constructing knowledge merchandise, not sustaining tables.
- Open Catalog (Powered by Apache Polaris): Dremio co-founded Apache Polaris, the open Iceberg catalog normal now graduated to a top-level Apache undertaking. Constructed on Polaris, Dremio’s Open Catalog supplies an Iceberg catalog that helps full learn and write from any REST-compatible engine, together with Spark, Flink, Trino, and DuckDB, all sharing the identical Iceberg tables. Governance, together with RBAC, row-level filters, column masking, and just-in-time credential merchandising, is enforced constantly on the catalog layer no matter which engine is querying. Each Dremio-managed desk is accessible to any Iceberg-compatible engine.
- Ingestion and Transformation: Dremio helps the complete vary of DML operations on Iceberg tables utilizing normal SQL. Steady ingestion by way of CREATE PIPE, batch masses by way of COPY INTO, and dbt Core integration make Dremio an entire platform for constructing and sustaining Iceberg-native knowledge pipelines.
Study extra about Dremio’s Iceberg capabilities at https://www.dremio.com/platform/apache-iceberg/