Data’s Grand Tour:
Navigating the Modern Data Architecture Maze
Lake, Data Warehouse, Mesh, Lakehouse, Fabric & Table Formats
Duration: 45 minutes | Date: 19th Sep 2025
Speaker: Aakash Khandelwal & Jyotika Agarwal
Flo 2025
The Data Challenge We Face
- 90% of the world’s data created in the last 2 years
- Organizations manage 10× more data types than 5 years ago
- Traditional architectures buckle under modern requirements
Today`s Journey
- Data Jargon 101
- 6 Core Architectures
- Comparative Analysis
- Decision Framework
- Table Formats Revolution
Data Jargon 101
Flo 2025 | 19th Sep 2025
Schema & Storage Models
- Schema-on-Read: Structure applied at query time for flexible ingestion.
- Schema-on-Write: Structure enforced at load time for consistent BI performance.
- Row-Oriented Storage: Records stored row by row for OLTP workloads.
- Columnar Storage: Columns stored together for OLAP queries, better compression.
- In-Memory Storage: Data in RAM for ultra-low latency analytics.
Data Jargon 101
Flo 2025 | 19th Sep 2025
Processing Patterns
- ETL (Extract, Transform, Load): Batch process — extract, cleanse, load into DWH.
- ELT (Extract, Load, Transform): Load raw data then transform in-place (lakehouse).
- Batch vs Streaming: Scheduled chunks vs real-time events.
Data Jargon 101
Flo 2025 | 19th Sep 2025
Access & Governance
- OLTP: Online Transaction Processing for high-volume writes.
- OLAP: Online Analytical Processing for complex read queries.
- Data Virtualization: Unified, virtual views without copying data.
- Metadata & Data Catalog: Searchable repository of schemas, lineage, quality.
Data Jargon 101
Flo 2025 | 19th Sep 2025
Transaction & Consistency
- ACID Transactions: Atomicity, Consistency, Isolation, Durability for reliable updates.
- Idempotent Processing: Repeatable transforms yield same result for exactly-once jobs.
Data Jargon 101
Flo 2025 | 19th Sep 2025
Advanced Concepts
- Data Lineage: Full lifecycle tracking from origin to report for auditability.
- Data Contract: Schema & SLAdefinitions between producers & consumers.
- Medallion Architecture: Bronze → Silver → >Gold
- Polyglot Persistence: Multiple storage technologies for workload-fit solutions.
Data Jargon 101
Flo 2025 | 19th Sep 2025
Jargons Summary
| Term | Definition |
|---|---|
| Schema-on-Read/Write | When structure is applied - read vs write |
| Columnar vs Row | Storage orientation - columns vs rows |
| ETL vs ELT | Order of transform vs load - before vs after |
| Streaming vs Batch | Real-time vs scheduled |
| Data Virtualization | Virtual view without movement |
| ACID Transactions | Reliable updates with guarantees |
| Data Lineage | Track data path from source to destination |
| Medallion Architecture | Layered Bronze/Silver/Gold |
Architecture Patterns
Flo 2025 | 19th Sep 2025
Pattern Deep Dive
Traditional
- ETL-first approach, Structured data warehouse, Batch processing focus
Lake-First
- Raw data ingestion, Schema-on-read flexibility, Cost-effective storage
Source: James Serra - "Deciphering Data Architectures," 2025
Architecture Patterns
Flo 2025 | 19th Sep 2025
Pattern Deep Dive
Hybrid
- Best of both worlds, Multiple processing engines, Workload optimization
Modern Unified
- Single platform approach, Real-time capabilities, Integrated governance
Architecture Patterns
Flo 2025 | 19th Sep 2025
When to Use Each Pattern
| Pattern | Best For | Key Benefit |
|---|---|---|
| Traditional | Structured reporting, compliance | Proven reliability |
| Lake-First | Diverse data types, exploration | Maximum flexibility |
| Hybrid | Mixed workloads, gradual migration | Balanced approach |
| Modern Unified | Real-time analytics, single platform | Simplified operations |
Data Architecture - Data Lake
Flo 2025 | 19th Sep 2025
Data Lake - Overview
A Data Lake ingests raw data of any format into a low-cost object store with schema-on-read flexibility. It serves as the central repository for batch and streaming data.
Source: DQlabs
Data Architecture - Data Lake
Flo 2025 | 19th Sep 2025
Data Organization
- Landing Zone: Batch (ADF, Glue) & Streaming (Kafka, Event Hubs)
- Raw Zone (Bronze): Immutable files (Parquet, ORC, JSON, CSV, multimedia)
- Curated Zone (Silver): Cleaned, conformed views/tables
- Business Zone (Gold): Aggregated, business-ready datasets
- Data Catalog: Hive Metastore, Glue Catalog, Azure Purview
Data Architecture - Data Lake
Flo 2025 | 19th Sep 2025
Patterns
- Bronze→Silver→Gold medallion layers
- Schema-on-Read flexibility
- Batch & Streaming integration
- Incremental Loads via partitions
- Sandboxing for prototyping
Data Architecture - Data Lake
Flo 2025 | 19th Sep 2025
Day-to-Day Operations
- Staging Offload: ETL staging on object storage
- Sandbox Provisioning: Quick Spark/Presto clusters
- Batch Window: Scheduled off-hours loads
- Raw Data Retention: Unlimited storage
- Incremental Loads: Partitioned folders
- Swamp Prevention: Metadata tagging & retention
Data Architecture - MDW
Flo 2025 | 19th Sep 2025
Modern Data Warehouse - Overview
A Modern Data Warehouse combines data lake and relational store capabilities with schema-on-write governance. It delivers optimized BI and reporting performance.
Source: Microsoft Learn
Data Architecture - MDW
Flo 2025 | 19th Sep 2025
Data Organization
- Staging Layer: ETL via SSIS, Informatica
- Raw Zone: Archive in data lake
- Curated Zone: Star/snowflake schemas
- Data Marts: Departmental subsets
- Semantic Layer: Business-friendly BI models
Data Architecture - MDW
Flo 2025 | 19th Sep 2025
Patterns
- ETL Orchestration scheduled workflows
- Delta Loads incremental data refresh
- Data Partitioning by date, region
- Slowly Changing Dimensions history tracking
- Semantic Views abstract complexity
Data Architecture - MDW
Flo 2025 | 19th Sep 2025
Day-to-Day Operations
- Maintenance Windows: Off-peak ETL runs
- Performance Tuning: Indexes & partitions
- Workload Management: Reporting vs loading queues
- Lineage Tracking: End-to-end audit trails
- Self-Service BI: Semantic layers for users
- Access Control: Single security boundary
Data Architecture - Data Fabric
Flo 2025 | 19th Sep 2025
Data Fabric - Overview
A Data Fabric provides real-time, virtualized access to distributed data sources without physical movement. It ensures governance via metadata-driven automation.
Source: Gartner
Data Architecture - Data Fabric
Flo 2025 | 19th Sep 2025
Data Organization
- Virtual Views: Logical tables over SQL, NoSQL, file stores
- Metadata Layer: Cataloging, lineage, data profiles
- API Gateway: Standardized REST/GraphQL endpoints
- Streaming Bus: Real-time connectors (Kafka, JMS)
- Policy Engine: Central security & masking rules
Data Architecture - Data Fabric
Flo 2025 | 19th Sep 2025
Patterns
- Virtualization real-time integration
- Metadata-Driven automation
- API-First data products
- Federated Policy enforcement
- Cache Acceleration for performance
Data Architecture - Data Fabric
Flo 2025 | 19th Sep 2025
Day-to-Day Operations
- Real-Time Queries: On-demand federation
- Metadata Harvesting: Auto-catalog updates
- Policy Application: Runtime masking & access control
- Cache Management: Accelerate frequent queries
- API Monitoring: Data product usage metrics
Data Architecture - Data Lakehouse
Flo 2025 | 19th Sep 2025
Data Lakehouse - Overview
A Data Lakehouse unifies data lake flexibility and data warehouse reliability with ACID transactions on open table formats.
Source: Databricks
Data Architecture - Data Lakehouse
Flo 2025 | 19th Sep 2025
Data Organization
- Bronze Layer: Raw ingests (append-only)
- Silver Layer: Cleansed & conformed tables
- Gold Layer: Aggregated, business-ready tables
- Table Formats: Delta Lake, Iceberg, Hudi
- Metadata Catalog: Hive Metastore or catalog store
Data Architecture - Data Lakehouse
Flo 2025 | 19th Sep 2025
Patterns
- ACID Transactions reliable updates
- Medallion Layers Bronze→Silver→Gold
- Unified Workloads BI, ML, streaming
- Compaction small-file optimization
- Schema Evolution add/rename columns
Data Architecture - Data Lakehouse
Flo 2025 | 19th Sep 2025
Day-to-Day Operations
- Acid Operations: Updates/deletes via SQL
- Compaction Jobs: Background file merging
- Time Travel Queries: Debug historical data
- Schema Evolution: Online adds/renames
- Hybrid Workloads: Mix batch, streaming, BI
Data Architecture - Data Mesh
Flo 2025 | 19th Sep 2025
Data Mesh - Overview
Data Mesh decentralizes data ownership to domain teams, treating data as a product under a federated governance framework.
Source: Estuary
Data Architecture - Data Mesh
Flo 2025 | 19th Sep 2025
Data Organization
- Domain Data Products: Autonomous datasets owned by teams
- Self-Serve Platform: Shared infrastructure (Kubernetes, Terraform)
- Data Contracts: Schema, SLA, quality agreements
- Federated Governance: Global policies enforced via code
- Discovery Catalog: Central registry of domain products
Source: Kelton Tech
Data Architecture - Data Mesh
Flo 2025 | 19th Sep 2025
Patterns
- Domain Ownership teams own their data
- Data as Product SLA-driven deliverables
- Federated Governance global standards
- Self-Serve Infra reusable components
- Contract Interfaces clear schemas & APIs
Data Architecture - Data Mesh
Flo 2025 | 19th Sep 2025
Day-to-Day Operations
- Domain Pipelines: Teams manage ETL schedules
- Product SLAs: Monitor freshness, quality, usage
- Platform Templates: Bootstrap new domains rapidly
- Contract Interfaces: Enforce schema & API standards
- Cross-Domain Discovery: Central catalog for products
Data Architecture - Microsoft Fabric
Flo 2025 | 19th Sep 2025
Microsoft Fabric - Overview
Microsoft Fabric is a unified lakehouse platform on OneLake, integrating Data Factory, Synapse, Power BI, and Notebooks with Direct Lake analytics.
Source: Microsoft
Data Architecture - Microsoft Fabric
Flo 2025 | 19th Sep 2025
Data Organization
- OneLake: Tenant-wide Delta Parquet store
- Workloads: Data Factory, Synapse SQL/Spark, Power BI, Notebooks
- Direct Lake Mode: Live Power BI on Delta tables
- Governance: MIP & DLP policies across workloads
- Domain Workspaces: Governed self-service areas
Source: Microsoft
Data Architecture - Microsoft Fabric
Flo 2025 | 19th Sep 2025
Key Components
- OneLake Storage: Delta Parquet foundation
- Data Factory: Managed pipelines and integration
- Synapse Pools: SQL & Spark compute
- Power BI: Direct Lake dashboards
- Notebooks: Data science & analytics
Data Architecture - Microsoft Fabric
Flo 2025 | 19th Sep 2025
Patterns
- Direct Lake live analytics
- Git Integration version control
- Auto Lineage cross-workload tracing
- Auto-Scale workload-based compute
- Workspace Isolation domain-specific environments
Data Architecture - Microsoft Fabric
Flo 2025 | 19th Sep 2025
Day-to-Day Operations
- Data Refresh: Automated pipelines via Data Factory
- Direct Lake Dashboards: Live updates without import
- Notebook CI/CD: Git-backed development
- Lineage Insights: Auto-generated lineage views
- Capacity Monitoring: Auto-scale compute usage
Comparative Analysis
Flo 2025 | 19th Sep 2025
Architecture Comparison Matrix
| Architecture | Schema | Cost | Performance | Complexity |
|---|---|---|---|---|
| Data Lake | Schema-on-Read | Very Low | Variable | Low |
| Data Warehouse | Schema-on-Write | High | Predictable | Medium |
| Data Lakehouse | Hybrid | Medium | High | Medium |
| Data Mesh | Federated | Variable | Scalable | High |
| Data Fabric | Virtual | Medium | Variable | High |
Comparative Analysis
Flo 2025 | 19th Sep 2025
Data Architectures
Source: Databricks
Comparative Analysis
Flo 2025 | 19th Sep 2025
Decision Framework - Part 1
Choose Data Lake When:
- ✅ Need cost-effective storage
- ✅ Handling diverse data types
- ✅ Exploratory analytics & ML
- ✅ Rapid data ingestion required
Choose Data Warehouse When:
- ✅ Need consistent BI performance
- ✅ Structured reporting requirements
- ✅ Strong governance needed
- ✅ Predictable query patterns
Comparative Analysis
Flo 2025 | 19th Sep 2025
Decision Framework - Part 2
Choose Lakehouse When:
- ✅ Need unified platform for BI & ML
- ✅ ACID transactions required
- ✅ Real-time & batch workloads
- ✅ Open format preference
Choose Data Mesh When:
- ✅ Large distributed teams
- ✅ Domain-specific expertise
- ✅ Organizational scalability
- ✅ Decentralized ownership
Table Formats Revolution
Flo 2025 | 19th Sep 2025
The Missing Link
Remember our architectures? Each had trade-offs. Table formats solve the fundamental problem of bringing warehouse reliability to lake economics.
Traditional Data Lake Problems:
- ❌ No ACID transactions
- ❌ Inconsistent reads during writes
- ❌ No schema enforcement
- ❌ Difficult updates/deletes
- ❌ Small files performance issues
Table Formats Revolution
Flo 2025 | 19th Sep 2025
Table Formats Solution
- ✅ ACID guarantees on object storage
- ✅ Consistent snapshot isolation
- ✅ Schema evolution & enforcement
- ✅ Efficient updates/deletes/merges
- ✅ Automatic compaction & optimization
Table Formats Revolution
Flo 2025 | 19th Sep 2025
ACID Transactions Explained
- Atomicity: All-or-nothing operations
Example: INSERT 1000 records - either all succeed or all fail - Consistency: Data integrity maintained
Example: Foreign key constraints enforced across updates - Isolation: Concurrent operations don't interfere
Example: Reader sees complete snapshot, not partial writes - Durability: Committed changes persist
Example: System crash after commit - data remains intact
Table Formats Revolution
Flo 2025 | 19th Sep 2025
Modern Table Formats - Overview
These formats enable Lakehouse architecture by bringing warehouse capabilities to data lakes.
Delta Lake
Databricks-originated, Spark-native
Apache Iceberg
Netflix-created, engine-agnostic
Apache Hudi
Uber-developed, streaming-focused
Table Formats Revolution
Flo 2025 | 19th Sep 2025
Delta Lake - Key Features
- ACID Transactions: Reliable concurrent reads/writes
- Time Travel: Query historical versions
- Schema Evolution: Add/modify columns safely
- Optimizations: Z-ordering, compaction
- Data Quality: Constraints & expectations
Table Formats Revolution
Flo 2025 | 19th Sep 2025
Delta Lake - Best Practices
- 🔧 Use partitioning for large tables
- 🔧 Regular OPTIMIZE operations
- 🔧 Set appropriate retention policies
- 🔧 Leverage Z-ORDER for co-location
- 🔧 Monitor small files problem
Table Formats Revolution
Flo 2025 | 19th Sep 2025
Table Formats Comparison
| Feature | Delta Lake | Apache Iceberg | Apache Hudi |
|---|---|---|---|
| ACID Support | Full | Full | Partial |
| Time Travel | Yes | Yes | Limited |
| Engine Support | Spark-focused | Multi-engine | Spark-focused |
| Streaming | Excellent | Good | Excellent |
| Maturity | High | Growing | Medium |
Choose based on your ecosystem: Delta for Databricks, Iceberg for multi-engine, Hudi for streaming-heavy workloads
Thank You
Flo 2025 | 19th Sep 2025
Thank You!
Questions & Discussion
Reference Site
https://aakashkh.github.io/data-grand-tour/
Scan the QR code or visit the link for additional resources, examples, and deep-dive content
Speakers: Aakash Khandelwal & Jyotika Agarwal
Data's Grand Tour: Navigating the Modern Data Architecture Maze