Real-Time Data Architecture: Streaming at Scale

Real-time data architecture answers a fundamental business question: “What is happening right now?” — fraud happening in the last 30 seconds, system anomalies in the last 5 minutes, inventory changes in the last hour. This guide covers how to build systems that detect, process, and serve insights from data as it is generated.

What Is Real-Time?

“Real-time” means different things to different teams. It’s important to be precise:

Most “real-time” business requirements are actually near-real-time — 5 to 15 minute latency is usually acceptable. True millisecond streaming is reserved for fraud, anomaly detection, and safety-critical systems.

Core Components of a Streaming Architecture

PRODUCERS                MESSAGE BROKER           PROCESSORS           CONSUMERS
─────────────────────    ─────────────────────    ─────────────────    ──────────────────────
Applications         →   Apache Kafka         →   Apache Flink     →   Real-Time Dashboards
IoT Devices          →   Azure Event Hubs     →   Spark Streaming  →   Alerts / Notifications
Databases (CDC)      →   AWS Kinesis          →   KQL Engine       →   Delta/Iceberg Tables
Clickstream          →   Google Pub/Sub       →   Kafka Streams    →   Operational Databases
Log streams          →   Confluent Cloud      →   Azure ASA        →   ML Inference Endpoints

The Message Broker: Apache Kafka

Kafka is the backbone of most real-time architectures. It is a distributed, durable, high-throughput event log.

Core Concepts

Topics & Partitions Events are published to named topics (e.g. order-events, clickstream). Each topic is split into partitions that enable parallelism. Consumers in a consumer group each read from a subset of partitions.

Topic: order-events
├── Partition 0: [event_1, event_4, event_7, ...]
├── Partition 1: [event_2, event_5, event_8, ...]
└── Partition 2: [event_3, event_6, event_9, ...]

Consumer Groups & Offsets Each consumer group tracks its offset (position in each partition) independently. This allows multiple applications to consume the same topic independently at their own pace:

• Flink job reads order-events to power real-time fraud detection
• Spark job reads the same order-events topic to write to the data lake
• Both are independent, neither blocks the other

Retention Unlike queues that delete messages after consumption, Kafka retains messages for a configurable period (hours, days, weeks). This enables replay — reprocessing historical events with new logic.

Kafka Managed Services

Stream Processing: Apache Flink

Apache Flink is the leading framework for stateful, event-time stream processing — processing events in the order they actually occurred (not in the order they were received), even when events arrive late.

Event Time vs Processing Time

Event timeline:    [09:00:01] [09:00:03] [09:00:04]  ← actual event times
Network delay:                                         ← events arrive out of order
Processing time:   [09:00:05] [09:00:03] [09:00:07]  ← arrival at Flink

Event time processing uses the timestamp embedded in the event (e.g. when the click happened) rather than when Flink received it. This is critical for accurate time-window aggregations.

Watermarks

Watermarks signal how far behind in event time Flink is willing to wait for late arrivals before closing a window:

Watermark = max_event_time_seen - allowed_lateness

If allowed_lateness = 10 seconds, Flink waits 10 seconds beyond the window close before emitting results, catching most late events.

Window Types

# Flink: count events per user in 5-minute tumbling windows
stream.key_by(lambda e: e.user_id) \
      .window(TumblingEventTimeWindows.of(Time.minutes(5))) \
      .aggregate(CountAggregate())

Stateful Operations

Flink maintains state per key across events — enabling:

• Running totals
• Session tracking
• Fraud pattern detection (e.g. “5 failed logins in 60 seconds”)
• Join streams over time (order matched with payment within 30 minutes)

State is stored in RocksDB (local) or in a remote state backend, checkpointed to object storage for fault tolerance.

Complex Event Processing (CEP)

CEP detects patterns across a sequence of events within a time window:

# Detect: login attempt followed by 3 failed password attempts within 60 seconds
pattern = Pattern.begin("login") \
    .where(lambda e: e.type == "LOGIN_ATTEMPT") \
    .next("failures") \
    .where(lambda e: e.type == "AUTH_FAILURE") \
    .times_or_more(3) \
    .within(Time.seconds(60))

Use cases:

• Fraud detection: unusual spending pattern following card swipe in new geography
• Intrusion detection: port scan followed by elevated privilege request
• Logistics: shipment departure not followed by arrival confirmation within SLA
• IoT: machine temperature spike followed by pressure drop = predicted failure

Spark Structured Streaming

Spark Structured Streaming treats a live data stream as an unbounded table — continuously appending rows. You write standard Spark SQL / DataFrame transformations, and Spark handles the incremental execution automatically.

# Read from Kafka
stream_df = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "broker:9092") \
    .option("subscribe", "order-events") \
    .load()

# Transform
orders = stream_df.select(
    from_json(col("value").cast("string"), schema).alias("data")
).select("data.*")

# Write to Delta Lake (streaming sink)
orders.writeStream \
    .format("delta") \
    .option("checkpointLocation", "/checkpoints/orders") \
    .outputMode("append") \
    .start("/mnt/delta/orders")

Trigger modes:

• trigger(processingTime="0 seconds"): process as fast as possible (continuous micro-batch)
• trigger(processingTime="1 minute"): process every minute
• trigger(once=True): process all available data once then stop (incremental batch)
• trigger(availableNow=True): process all available, respect rate limits

Real-Time Architecture Patterns

Pattern 1: Kafka → Flink → Delta Lake (Lakehouse Streaming)

Producers → Kafka → Flink/Spark Streaming → Delta Bronze table → dbt/Spark Silver/Gold → BI

Best for: unified streaming + batch on a single lakehouse. Real-time data in Delta Bronze within seconds; Gold tables refreshed on short schedules.

Pattern 2: Event-Driven Microservices (CQRS + Event Sourcing)

Service A publishes event → Kafka → Service B consumes event
                                 → Service C consumes same event
                                 → Analytics consumer writes to lake

Best for: microservice architectures where services communicate via events and analytics is a side-effect of operational events.

Pattern 3: Real-Time Dashboard (KQL Database)

Event Hubs / Kafka → EventStream → KQL Database (Fabric / Kusto)
                                         ↓
                                Real-Time Dashboard (sub-second queries)

Best for: operational dashboards, monitoring, telemetry analysis where sub-second query latency is required.

Pattern 4: Change Data Capture (CDC)

PostgreSQL / MySQL → Debezium (CDC connector) → Kafka → Flink → Data Lake

Debezium captures every row-level INSERT/UPDATE/DELETE from operational databases as events, enabling near-real-time replication to the data lake without polling.

Key Design Considerations

Exactly-Once vs At-Least-Once

Most production systems use at-least-once with idempotent writes — cheaper than full exactly-once while achieving the same observable outcome.

Backpressure

When a consumer processes events slower than the producer generates them, backpressure signals the producer to slow down. Flink handles backpressure automatically — Kafka acts as a durable buffer absorbing traffic spikes.

Late Data Handling

Events can arrive late due to network delays, mobile offline usage, or IoT connectivity issues. Design choices:

• Drop: ignore events arriving after window closes (simple, some data loss)
• Watermarks + allowed lateness: wait a configurable period before closing windows
• Side output: route late events to a separate stream for offline reconciliation

When to Invest in Real-Time Architecture

✅ Justified when:

• Business decisions are made on data that is minutes to hours old today — with measurable cost
• You have fraud, risk, or safety-critical detection requirements
• Operational dashboards drive real-time business actions (logistics, customer service)
• Your data volumes require streaming to avoid massive batch jobs

❌ Unjustified when:

• “Real-time” is aspirational but actual decisions are made daily — hourly batch is sufficient
• Your team lacks streaming expertise — operational complexity is high
• Data sources don’t generate continuous events — batch ingestion is the natural fit