iceberg

Apache Iceberg Expert

Scope

Iceberg table design, engine integration, catalog choice, schema and partition evolution, time travel, row-level operations, and maintenance.

For Flink job architecture use the flink skill, for Paimon streaming ingestion use paimon, and for Fluss hot storage use fluss. This skill applies to the Iceberg table those systems read or write, not to the engine itself.

Current Facts

• Current Apache Iceberg project release: 1.11.0, released May 19, 2026.
• Format versions: v1, v2, and v3 are complete and adopted by the Iceberg community.
• Format v2: production baseline for row-level deletes and broad engine compatibility.
• Format v3: adds nanosecond timestamp types, variant, geometry/geography, unknown type, default values, multi-argument transforms, row lineage, binary deletion vectors, and table encryption keys.
• Format v4: under active development and not formally adopted. The spec now names it "Metadata Structure and Representation", headlined by relative locations in metadata.
• Polaris: Apache Polaris graduated to a Top-Level Project in February 2026 and is a vendor-neutral Iceberg REST catalog implementation. Current release 1.6.0, July 9, 2026.
• Engine v3 support is uneven. Check the specific engine and the specific feature. Snowflake reached v3 GA on May 7, 2026, though external-engine writes through the Horizon REST Catalog are not yet supported. AWS has been GA since November 2025 but only for deletion vectors and row lineage, only on Spark-based services such as EMR 7.12+, Glue, and S3 Tables. Amazon Athena does not support v3. Do not describe v3 support as universally pending, and do not describe it as universal either.

Inspect First

Establish before recommending or changing anything:

1. Engine and version, and catalog type. Behaviour varies across Spark, Flink,

Trino, Athena, Snowflake, REST, Glue, Hive, Nessie, and Polaris.

1. Current format version, from the table's format-version property.
2. For maintenance work, read the snapshots, manifests, files, and

partitions metadata tables. Get snapshot count and age, manifest count, delete file and deletion vector counts, and the actual file size distribution rather than assuming a small-file problem.

1. Which other engines and jobs write to the table.

Decision Rules

• Choose v2 for maximum compatibility. Choose v3 only when every engine that

reads or writes the table supports the v3 features you need.

• Treat a v2 to v3 upgrade as a compatibility event, not a property edit.
• Use hidden partitioning and transform functions rather than exposing physical

partition columns to users.

• Match partition transforms to real query predicates. Over-partitioning

inflates metadata and slows planning more often than it speeds scans.

• Prefer deletion vectors over positional delete files where the engine

supports them; merge-on-read cost scales with delete file count.

• Compact after streaming or high-frequency writes, driven by the observed file

size distribution rather than a schedule alone.

Safety

• expire_snapshots permanently deletes data files unreachable from retained

snapshots. Before running it, confirm the retention window against rollback, audit, and time-travel requirements, state how many snapshots will be dropped, and get confirmation for a production table.

• remove_orphan_files deletes files not referenced by metadata, and files

being written by in-flight jobs look exactly like orphans. Keep older_than comfortably longer than the longest-running writer. The Spark procedure defaults to three days for this reason; do not lower it to reclaim space without first confirming no writers are active.

• DROP TABLE semantics differ by catalog. Some purge data, some only remove

the catalog entry. Confirm which applies before running it.

• Do not run maintenance against a production table without knowing every

engine that writes to it.

Verify

• After compaction, re-read files and report file count and average size

before and after, plus a row count showing data is unchanged.

• After snapshot expiry, report the snapshots remaining and confirm the oldest

retained snapshot still satisfies the stated time-travel requirement.

• After partition evolution, read partitions to confirm new writes land in

the new spec while existing data stays readable.

• After a v3 upgrade, run a read from every engine that touches the table.
• Report the engine and version you validated with, and name any engine you

could not test.

Update Checklist

• Recheck Apache Iceberg releases before changing library/runtime versions.
• Recheck the spec page before changing format-version wording.
• Recheck engine-specific v3 support before making upgrade recommendations.