The Data Lake was the first architecture to break free from the rigid schemas of traditional data warehouses. By storing raw data in its native format at virtually unlimited scale, it opened the door to exploratory analytics, machine learning, and multi-modal data processing.
Overview
A Data Lake ingests raw data of any format — structured tables, semi-structured JSON/XML, unstructured text/images/video — into a low-cost object store with schema-on-read flexibility. It serves as the central, single source of truth for all raw data across an organisation.
Data Organisation
Well-designed data lakes are organised into zones (sometimes called layers or hops) that represent the progressive refinement of data quality:
Landing / Ingestion Zone
The entry point for all raw data. Ingestion is triggered via:
- Batch: Azure Data Factory (ADF), AWS Glue, Apache NiFi, Fivetran
- Streaming: Apache Kafka, Azure Event Hubs, AWS Kinesis, Google Pub/Sub
Raw Zone (Bronze)
Immutable, append-only storage of data exactly as it arrived. File formats:
- Parquet: Columnar, compressed, widely supported — the default choice
- ORC: Optimised for Hive and Hadoop workloads
- JSON / CSV: For semi-structured and legacy ingestion
- Avro: Schema-embedded, ideal for Kafka event streams
- Multimedia: Images, audio, video for unstructured data pipelines
Nothing is ever deleted from the Bronze zone — it is the ground truth and enables full reprocessing.
Curated / Refined Zone (Silver)
Cleaned, validated, and deduped data ready for analytical consumption. Operations performed here:
- Null handling and deduplication
- Type casting and standardisation
- Joins with reference/dimension data
- Partitioning for query performance (by date, region, etc.)
Aggregated Zone (Gold)
Pre-aggregated, business-domain-specific datasets powering dashboards and ML models. Typically stored in columnar format optimised for BI tools.
Strengths
- Unlimited scale — object storage costs pennies per GB
- Schema flexibility — ingest without upfront modelling
- Multi-modal — structured, semi-structured, unstructured in one place
- Decoupled compute — scale storage and compute independently
- Full raw retention — Bronze zone enables any-time reprocessing
- ML / AI ready — direct access to raw data for model training
Limitations
- No ACID transactions — concurrent writes can corrupt data
- Poor query performance on raw data — no pre-computed aggregations
- Metadata complexity — without a catalog, lakes become "data swamps"
- No native update/delete — updating records requires full file rewrites
- Governance challenges — fine-grained access control is harder
- Data quality risk — without enforcement, quality degrades over time
Common Technology Stack
Cloud Object Stores
| Provider | Service |
|---|---|
| Microsoft Azure | Azure Data Lake Storage Gen2 (ADLS Gen2) |
| AWS | Amazon S3 |
| Google Cloud | Google Cloud Storage (GCS) |
Ingestion Tools
| Category | Tools |
|---|---|
| Batch ETL | Azure Data Factory, AWS Glue, Apache NiFi, Fivetran, Airbyte |
| Streaming | Apache Kafka, Azure Event Hubs, AWS Kinesis |
| CDC | Debezium, Attunity |
Query & Processing Engines
| Tool | Use Case |
|---|---|
| Apache Spark | Large-scale distributed batch and streaming transforms |
| Azure Databricks | Managed Spark with ML integration |
| Presto / Trino | Interactive SQL over object storage |
| AWS Athena | Serverless SQL over S3 |
| Azure Synapse Analytics | Integrated analytics over ADLS |
| Apache Hive | SQL abstraction over Hadoop/HDFS |
When to Choose a Data Lake
✅ Good fit when:
- You need to ingest diverse, unstructured data (logs, clickstreams, images, IoT telemetry)
- You require schema flexibility and don’t want to model data upfront
- You are building ML/AI pipelines that need access to raw historical data
- Cost is a primary concern (object storage is 10-20× cheaper than managed warehouses)
- You need to retain all historical data for compliance or reprocessing
❌ Poor fit when:
- Your primary use case is BI reporting with strict SLAs on query performance
- You need ACID transactions or consistent updates at record level
- Your team lacks the engineering capacity to manage zone organisation and metadata
- Data governance and access control are critical requirements from day one
Evolution: From Data Lake to Lakehouse
The pure Data Lake model struggled with governance, ACID compliance, and BI performance. This drove the evolution toward the Data Lakehouse — which adds a transactional metadata layer (Delta Lake, Apache Iceberg, Apache Hudi) on top of the data lake to deliver warehouse-grade reliability while retaining lake-scale flexibility.