Apache Iceberg

On this page we explore Apache Iceberg, including its main benefits and how IOMETE leverages this modern open source table format.

Note: An extensive version of the information below can be found on the website of the Apache Iceberg organization.

What is Apache Iceberg?

Iceberg is a high-performance open table format for huge analytic datasets. Iceberg brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables.

Open Standard

Iceberg has been designed and developed to be an open community standard to ensure compatibility across languages and implementations. Apache Iceberg is open source, and is developed at the Apache Software Foundation.

IOMETE and Apache Iceberg

IOMETE is a fully-managed (ready to use, batteries included) data platform. IOMETE optimizes clustering, compaction, and access control to Iceberg tables. The core of the IOMETE platform is a serverless lakehouse that leverages Apache Iceberg as its core table format.

Apache Iceberg Benefits

Iceberg avoids unpleasant surprises. Schema evolution works and won’t inadvertently un-delete data. Users don’t need to know about partitioning to get fast queries.

Schema evolution supports add, drop, update, or rename, and has no side effects.
Hidden partitioning prevents user mistakes that cause silently incorrect results or extremely slow queries.
Partition layout evolution can update the layout of a table as data volume or query patterns change.
Time Travel enables reproducible queries that use exactly the same table snapshot, or lets users easily examine changes.

Reliability and Performance

Iceberg was built for huge tables. Iceberg is used in production where a single table can contain tens of petabytes of data and even these huge tables can be read without a distributed SQL engine.

Scan planning is fast - A distributed SQL engine isn’t needed to read a table or find files.
Advanced filtering - Data files are pruned with partition and column-level stats, using table metadata.
Iceberg was designed to solve correctness problems in eventually-consistent cloud object stores.
Works with any cloud store and reduces NN congestion when in HDFS, by avoiding listing and renames.
Multiple concurrent writers use optimistic concurrency and will retry to ensure that compatible updates succeed, even when writes conflict.
Serializable isolation - table changes are atomic and readers never see partial or uncommitted changes.