Table Formats Revolution: Delta Lake, Iceberg & Hudi

Open table formats are the technological revolution at the heart of the modern data lakehouse. They transform simple object storage (ADLS, S3, GCS) — which is fundamentally immutable and lacks any concept of transactions — into reliable, queryable, version-controlled databases.

Why Table Formats Exist

Object storage stores files. That’s all. It has no concept of:

• Transactions: two writers simultaneously can overwrite each other
• Schema: there’s no enforced structure; a CSV and a Parquet file look the same to S3
• Versions: you can’t query “what did this data look like last Tuesday?”
• Updates: you can’t change one row in a Parquet file without rewriting the entire file

Table formats solve all of this by adding a metadata layer — a structured set of logs and manifests that track every change to the dataset, building a complete history of the table’s state.

Delta Lake

Architecture

Delta Lake stores table data as Parquet files in object storage, alongside a _delta_log/ directory containing a sequence of JSON transaction log entries.

my-table/
├── _delta_log/
│   ├── 00000000000000000000.json   ← initial snapshot
│   ├── 00000000000000000001.json   ← first write
│   ├── 00000000000000000002.json   ← second write
│   └── 00000000000000000010.checkpoint.parquet  ← periodic checkpoint
├── part-00000-abc.snappy.parquet
├── part-00001-def.snappy.parquet
└── ...

Each log entry records:

• add: new Parquet files added
• remove: files logically deleted (not physically removed until VACUUM)
• metadata: schema changes, table properties
• commitInfo: operation type, timestamp, userId

Key Capabilities

ACID Transactions Concurrent writers use optimistic concurrency control — each writer records its changes in a new log entry. If two writers conflict, one fails with a concurrency error and retries. Readers always see a consistent snapshot.

Time Travel

-- Query by version number
SELECT * FROM my_table VERSION AS OF 42;

-- Query by timestamp
SELECT * FROM my_table TIMESTAMP AS OF '2025-09-01 00:00:00';

-- Restore to previous version
RESTORE TABLE my_table TO VERSION AS OF 10;

Schema Evolution

# Add new columns without rewriting data
df.write.option("mergeSchema", "true").format("delta").mode("append").save(path)

Z-Order Clustering Z-Order is a multi-dimensional space-filling curve that co-locates related data in the same Parquet files. If you frequently filter on (country, date), Z-ordering on those columns means queries skip most files:

OPTIMIZE my_table ZORDER BY (country, date);

This reduces data scanned — queries that previously read 10TB now read 100GB.

Liquid Clustering (Delta 3.0+) An improvement over Z-Order that uses a bloom filter-based incremental clustering algorithm:

• Runs incrementally — no full table rewrite
• Self-manages as data grows
• Better for high-cardinality columns

-- Enable liquid clustering at table creation
CREATE TABLE my_table (id INT, country STRING, date DATE)
CLUSTER BY (country, date);

-- Trigger clustering on new data only
OPTIMIZE my_table;

VACUUM The VACUUM command physically deletes Parquet files that are no longer referenced by any live table version (older than the retention period):

VACUUM my_table RETAIN 168 HOURS;  -- keep 7 days of history

Apache Iceberg

Architecture

Iceberg uses a tree-structured metadata hierarchy that enables extremely efficient metadata operations at scale:

metadata/
└── v3.metadata.json           ← current metadata file
    ├── schema
    ├── partition-spec
    └── snapshot-id → ─────────────────────────────────────────────→ snapshots/
                                                                      └── snap-123.avro
                                                                          └── manifest list
                                                                              └── manifests/
                                                                                  └── manifest-abc.avro
                                                                                      └── data files
data/
└── country=US/date=2025-09/
    └── part-00001.parquet

This separation means metadata operations (listing what files changed) don’t require scanning all data files — a huge advantage for tables with billions of files.

Key Capabilities

Hidden Partitioning With Iceberg, you don’t need to know the partition structure to query efficiently. Iceberg applies partition transforms transparently:

-- Table partitioned by day(event_time), but queries don't need to filter on a partition column
SELECT * FROM events WHERE event_time BETWEEN '2025-09-01' AND '2025-09-30';
-- Iceberg automatically resolves this to the correct partitions

Partition Evolution Change partitioning strategy without rewriting any data:

-- Old partition: by month
ALTER TABLE events ADD PARTITION FIELD day(event_time);
-- Future data uses day partitioning; old data remains month-partitioned
-- Both are transparently queryable together

Row-Level Deletes (Equality Deletes) Instead of rewriting entire Parquet files to delete rows, Iceberg tracks deletes in separate delete files:

• Position deletes: record the file path + row position of deleted rows
• Equality deletes: record the values of deleted rows — any row matching those values is excluded at read time

This makes CDC (Change Data Capture) and GDPR deletion requests dramatically cheaper.

Snapshot Isolation Every write creates a new immutable snapshot. Readers always operate on a consistent snapshot, even during concurrent writes.

-- Travel to any historical snapshot
SELECT * FROM events FOR VERSION AS OF 123456789;

Apache Hudi

Architecture

Hudi maintains a timeline of all actions performed on the table, stored in a .hoodie/ directory:

my-hudi-table/
├── .hoodie/
│   ├── 20250919123000.commit      ← completed commit
│   ├── 20250919123000.commit.requested
│   ├── 20250919123000.inflight
│   └── hoodie.properties
├── country=US/
│   └── part-20250919.parquet
└── ...

Storage Types

Copy-on-Write (CoW) Every update rewrites the affected Parquet files entirely. This is read-optimised — reads are simple sequential Parquet scans with no merge overhead. But writes are expensive for high-update workloads.

Merge-on-Read (MoR) Updates are written as delta log files alongside base Parquet files. At read time, delta logs are merged with the base files. This is write-optimised — fast incremental ingestion. A periodic compaction job merges deltas into base files to maintain read performance.

Base file (Parquet):  [id=1, name="Alice", version=1]
Delta log:            [id=1, name="Alice Updated", version=2]
At read: merged →    [id=1, name="Alice Updated", version=2]

Incremental Queries

Hudi’s killer feature — query only records that changed since a given timestamp:

# Read only records committed after a specific instant
incremental_df = spark.read.format("hudi") \
    .option("hoodie.datasource.query.type", "incremental") \
    .option("hoodie.datasource.read.begin.instanttime", "20250918000000") \
    .load(base_path)

This enables efficient incremental ETL pipelines — instead of reprocessing all data daily, only process what changed.

Comparison: Delta Lake vs Iceberg vs Hudi

Compaction & Maintenance

All three formats require periodic maintenance operations:

Small File Problem

Streaming ingestion creates many small Parquet files (each micro-batch = new files). Small files hurt query performance because:

• More files = more metadata overhead
• Parallelism is less efficient on tiny files
• Object storage LIST operations are expensive

Solution — Compaction:

-- Delta Lake: merge small files into larger ones
OPTIMIZE my_table;

-- Iceberg: rewrite files via Spark
CALL catalog.system.rewrite_data_files('my_table');

Vacuuming / Expiring Snapshots

Old files that are no longer referenced by live snapshots must be physically deleted:

-- Delta Lake
VACUUM my_table RETAIN 168 HOURS;

-- Iceberg: expire old snapshots
CALL catalog.system.expire_snapshots('my_table', TIMESTAMP '2025-09-01 00:00:00');

-- Hudi: clean old versions
hoodie.cleaner.commits.retained = 10

Choosing a Table Format