Architecture Patterns: Medallion, Lambda, Kappa & Beyond

Understanding individual architectures (Lake, Warehouse, Lakehouse) is one thing — but how do you actually pattern your data flows within them? This article covers the recurring design patterns that appear across every modern data platform.

Medallion Architecture (Multi-Hop)

The Medallion Architecture (also called the Multi-Hop or Bronze-Silver-Gold pattern) organises data into progressive quality tiers, with each tier adding more refinement, validation, and business context.

The Three Zones

SOURCES                 BRONZE              SILVER              GOLD
─────────────────────   ─────────────────   ─────────────────   ──────────────────
Operational DBs    ──→  Raw, immutable   →  Cleaned, joined  →  Aggregated, curated
APIs               ──→  Schema-on-read      Schema-on-write     Business domains
Streaming events   ──→  Full history        Validated           ML features
Files/Uploads      ──→  No deletions        Deduplicated        Reporting marts

Bronze (Raw) Layer

Purpose: Capture data exactly as it arrives. Nothing is modified, filtered, or discarded.

Characteristics:

• Immutable append-only writes
• Retains original data types, nulls, duplicates
• Partitioned by ingestion date (not event date)
• Format: Delta Lake / Iceberg tables (or raw Parquet/JSON if pre-table-format)
• Schema: minimal enforcement — absorb schema drift

Operations: Ingestion only (ADF, Autoloader, Kafka → Delta)

Silver (Curated) Layer

Purpose: Clean, validate, and join data to make it analytically reliable.

Transformations applied:

• Null handling: drop or impute missing values
• Deduplication: remove duplicate events (idempotent processing)
• Type standardisation: parse strings to proper date/int types
• Joins: enrich with reference data (country codes, product names)
• Filtering: remove test data, system noise, invalid records
• Partitioning: re-partition by business date for query efficiency

Schema: enforced, documented, tested with Great Expectations / dbt tests

Gold (Business) Layer

Purpose: Deliver pre-aggregated, domain-specific datasets for dashboards and ML models.

Characteristics:

• Aggregate metrics by business dimensions (daily sales by region and product)
• Denormalised for fast BI access (fewer joins at query time)
• Semantic naming aligned with business terminology
• Served directly to Power BI, Tableau, Looker
• Updated on a schedule matching business SLAs (hourly, daily)

Lambda Architecture

The Lambda Architecture was the dominant pattern before lakehouse unified streaming and batch. It runs two separate processing paths for different latency requirements.

DATA SOURCE
     │
     ├──────────────→ BATCH LAYER (Spark, Hadoop)
     │                 - Reprocesses all historical data
     │                 - Produces accurate, complete results
     │                 - High latency (hours/days)
     │
     └──────────────→ SPEED LAYER (Flink, Storm, Kafka Streams)
                       - Processes real-time events
                       - Approximate or partial results
                       - Low latency (seconds)
                            │
                            ↓
                    SERVING LAYER (merges both)
                    - Queries combine batch views + speed views
                    - Presents unified answer to consumers

Strengths

• Provides both real-time and accurate historical results
• Batch layer is the source of truth — speed layer is a temporary approximation

Limitations

• Dual codebase: same logic must be maintained in two systems (Spark + Flink)
• Complexity: merging batch and speed views at query time is error-prone
• Operational overhead: two separate infrastructures to monitor and maintain
• Largely superseded by the Kappa Architecture and unified streaming platforms

Kappa Architecture

The Kappa Architecture simplifies Lambda by eliminating the batch layer — treating everything as a stream.

DATA SOURCE
     │
     └──────────────→ STREAMING LAYER (Kafka + Flink / Spark Streaming)
                       - All data is processed as events
                       - Reprocessing = replay Kafka topic from offset 0
                       - Single codebase handles both real-time and historical
                            │
                            ↓
                    SERVING LAYER
                    - Materialized views in a database or data lake
                    - Same data, same logic, simpler operations

Kappa in Practice

The key insight: if you store raw events in Kafka with a long retention (days, weeks, months), you can reprocess historical data simply by replaying from the beginning of the topic. No separate batch job needed.

Strengths

• Single codebase — no Lambda dual-maintenance problem
• Simpler operations and debugging
• Kafka topic is the ground truth

Limitations

• Long Kafka retention is expensive at petabyte scale
• Reprocessing from scratch is slow for very long historical datasets
• Less suitable when batch reprocessing is done infrequently over years of data

When Lambda vs Kappa

Data Layout Optimisation Patterns

Partitioning

Partitioning physically separates data into directories by a column value. Queries that filter on the partition column skip entire directories — reducing data scanned dramatically.

events/
├── date=2025-09-01/
│   └── part-00000.parquet
├── date=2025-09-02/
│   └── part-00000.parquet
└── ...

Best practices:

• Choose low-to-medium cardinality columns (date, country, status — not user_id)
• Avoid over-partitioning: 10,000+ partitions with tiny files is worse than no partitioning
• Typical partition columns: date, year/month, region, event_type

Z-Order (Delta Lake)

Z-Order is a multi-dimensional co-location technique. Files are written such that records with similar values in the Z-ordered columns are stored together in the same Parquet file.

OPTIMIZE orders ZORDER BY (customer_id, order_date);

This enables file skipping — if a query filters on customer_id = 12345, only a tiny subset of files need to be opened. Z-Order is applied on top of (not instead of) partitioning.

Liquid Clustering (Delta 3.0+)

Liquid Clustering supersedes Z-Order with a more intelligent, incremental approach:

-- Define clustering columns at table creation
CREATE TABLE orders CLUSTER BY (customer_id, order_date);

-- Delta automatically clusters new data; run OPTIMIZE periodically
OPTIMIZE orders;

Advantages over Z-Order:

• Incremental: only newly written files are reorganised, not the entire table
• Automatic: self-tunes as query patterns evolve
• No full rewrite: doesn’t require a full table scan on every optimization run

Bloom Filters

Bloom filters are probabilistic data structures stored in Parquet file footers. They provide fast “is this value definitely NOT in this file?” lookups — enabling additional file skipping beyond partitioning and Z-ordering.

-- Enable bloom filter on high-cardinality lookup columns
ALTER TABLE orders SET TBLPROPERTIES (
  'delta.bloomFilter.columns' = 'order_id,transaction_id'
);

Data Vault

For completeness, Data Vault is a modelling methodology (not an architecture pattern per se) used inside data warehouses to enable highly auditable, historised, and source-system-agnostic data stores.

Data Vault is popular in highly regulated industries (banking, insurance) where complete audit trails and source traceability are mandatory. It is often used as the intermediate layer between raw and the Kimball star schema presentation layer.

Summary: Choosing a Pattern