The Data Lakehouse emerged as the architectural answer to a fundamental tension: data lakes offered unlimited scale and flexibility but lacked governance; data warehouses offered governance and performance but were rigid and expensive. The Lakehouse converges both into a single, unified platform.
Overview
A Data Lakehouse adds a transactional metadata layer on top of low-cost object storage (ADLS, S3, GCS) to deliver:
- Warehouse-grade ACID transactions and schema enforcement
- Lake-scale storage with support for all data types (structured, semi-structured, unstructured)
- Unified platform for SQL analytics, data science, ML training, and streaming
🏗️ Core Philosophy
One platform. One copy of data. Warehouse reliability at lake economics.The Architecture
┌──────────────────────────────────────────────────────────────┐
│ WORKLOADS │
│ BI / SQL Analytics │ Data Science / ML │ Streaming / RT │
└─────────────────────┬────────────────────────────────────────┘
│
┌─────────────────────▼────────────────────────────────────────┐
│ OPEN TABLE FORMAT LAYER │
│ Delta Lake │ Apache Iceberg │ Apache Hudi │
│ (ACID, Schema evolution, Time Travel, Versioning) │
└─────────────────────┬────────────────────────────────────────┘
│
┌─────────────────────▼────────────────────────────────────────┐
│ OBJECT STORAGE (Low-cost, Unlimited Scale) │
│ ADLS Gen2 │ Amazon S3 │ Google GCS │
└──────────────────────────────────────────────────────────────┘
The open table format layer is the key innovation. It adds a transaction log that tracks every change to the dataset, enabling ACID properties on top of otherwise immutable object storage.
Open Table Formats
Delta Lake
Originally developed at Databricks and now open-source under the Linux Foundation.
Key capabilities:
- ACID transactions via a
_delta_log/transaction log - Scalable metadata handling for tables with billions of files
- Time Travel: query any historical version of a table
- Z-Order Clustering: co-locate related data to speed up queries
- Liquid Clustering: automatic, intelligent data layout optimisation (newer)
- Schema enforcement and schema evolution
- Unified batch and streaming reads/writes (via Structured Streaming)
-- Time travel: query data as it was 7 days ago
SELECT * FROM events TIMESTAMP AS OF date_sub(current_date(), 7);
-- Restore to a previous version
RESTORE TABLE events TO VERSION AS OF 5;
Apache Iceberg
Developed by Netflix and Apple; now a top-level Apache project.
Key capabilities:
- Partition evolution: change partitioning strategy without rewriting data
- Hidden partitioning: queries don’t need to filter on partition columns explicitly
- Row-level deletes: efficient record-level updates/deletes using position-based delete files
- Full snapshot isolation and time travel
- Compatible with Spark, Flink, Trino, Dremio, and BigQuery
- Strong multi-engine support — excellent for polyglot environments
Apache Hudi
Developed by Uber; designed for incremental data pipelines with upsert-heavy workloads.
Key capabilities:
- Copy-on-Write (CoW): rewrites files on every update — optimal for read-heavy
- Merge-on-Read (MoR): appends delta logs and merges at read time — optimal for write-heavy
- Incremental query support — query only records changed since a given timestamp
- Built-in indexing for fast upserts (Bloom filter, HBase index)
- Native Spark and Flink integration
Comparison: Delta Lake vs Iceberg vs Hudi
| Feature | Delta Lake | Apache Iceberg | Apache Hudi |
|---|---|---|---|
| Primary Strength | Databricks ecosystem | Multi-engine compatibility | Incremental upserts |
| ACID Transactions | ✅ | ✅ | ✅ |
| Time Travel | ✅ | ✅ | ✅ |
| Schema Evolution | ✅ | ✅ (+ partition evolution) | ✅ |
| Row-level Updates | ✅ (MERGE) | ✅ (row-level deletes) | ✅ (native upsert) |
| Partition Evolution | Limited | ✅ Full | Limited |
| Streaming Support | ✅ | ✅ | ✅ |
| Best Ecosystem | Databricks, Spark | Flink, Trino, BigQuery | Spark, Kafka CDC |
| Governance / Cataloging | Unity Catalog | Iceberg Catalog (REST) | Hudi Metaserver |
Medallion Architecture in the Lakehouse
The Lakehouse naturally implements the Medallion Architecture:
| Layer | Storage | Content | Tools |
|---|---|---|---|
| Bronze | Delta/Iceberg tables | Raw ingested data, immutable | ADF, Autoloader, Kafka |
| Silver | Delta/Iceberg tables | Cleaned, validated, joined | Spark, dbt |
| Gold | Delta/Iceberg tables | Aggregated business metrics | dbt, Spark SQL |
💟 Production Note
The Medallion pattern is the universal starting point for any Lakehouse. Start with Bronze → Silver → Gold from day one, even if Gold is just a few basic aggregation tables. Retrofitting zone discipline onto an existing lake is significantly harder than establishing it upfront.Strengths
| Strength | Detail |
|---|---|
| ACID on Object Storage | Full transactional reliability without a proprietary warehouse engine |
| One Copy of Data | Eliminates the “lake-to-warehouse” ETL hop and its associated latency/cost |
| Unified Analytics | SQL, Python, Spark, ML — all on the same data |
| Schema Evolution | Add columns, change types without downtime or data rewrites |
| Time Travel | Audit, debug, and reproduce historical data states |
| Cost Efficiency | Open format on commodity object storage — no vendor lock-in |
| Streaming + Batch | Same table used for real-time ingestion and batch analytics |
Limitations
| Limitation | Impact |
|---|---|
| Small File Problem | Streaming ingestion creates many small files; requires compaction jobs |
| Metadata Overhead | Transaction logs grow with every write; requires periodic vacuuming |
| Operational Complexity | Managing compaction, vacuuming, and Z-ordering requires expertise |
| BI Tool Support | Some older BI tools don’t natively support Iceberg/Delta formats |
| Cold Query Performance | First queries on large tables can be slower without proper clustering |
Leading Platforms
| Platform | Table Format | Key Offering |
|---|---|---|
| Databricks | Delta Lake (native) | Unified analytics, MLflow, Unity Catalog |
| Apache Spark + Delta | Delta Lake | Open-source DIY lakehouse |
| Azure Synapse Analytics | Delta Lake + Iceberg | Integrated Azure ecosystem |
| AWS Lake Formation + Glue | Iceberg + Hudi | Managed AWS lakehouse |
| Snowflake | Iceberg (external) | Iceberg tables on S3/GCS with Snowflake compute |
| Google BigQuery | Iceberg (external) + BigLake | BigQuery over GCS with Iceberg |
| Dremio | Iceberg | Query engine with Iceberg-native Arctic catalog |
When to Choose a Lakehouse
✅ Good fit when:
- You want a single platform for BI, data science, and ML — without maintaining separate systems
- You need ACID guarantees but can’t afford the cost of storing everything in a managed warehouse
- Your data includes streaming + batch workloads on the same datasets
- You value open formats and want to avoid warehouse vendor lock-in
- You need schema evolution without painful migration scripts
- Your team is comfortable with Spark/Python alongside SQL
❌ Poor fit when:
- Your organisation is entirely SQL/BI focused with no engineering team to manage Spark and compaction
- You need sub-second query response for dashboards (managed warehouses still win here without tuning)
- You’re a small team needing a simple, managed solution with minimal operational overhead