Data Pipeline Architecture: Batch, Streaming, or Hybrid?

The right data pipeline architecture depends on how quickly your data must be actionable: if you need answers within seconds or minutes of data arrival, you need streaming or a hybrid approach; if you can wait hours or a day, batch is simpler and cheaper. This article explains the trade-offs so you can choose based on your latency, cost, and complexity tolerance.

Formula

The calculator above gives a rough suitability score (0-100) for batch versus streaming based on three inputs:

• Batch interval (t) – how often you currently run batch jobs, in hours.
• Streaming latency target (s) – the maximum acceptable delay between data generation and availability, in seconds.
• Real-time query ratio (r) – the percentage of queries or decisions that need up-to-the-minute data.

The batch score is 100 - min(100, (s/60) + (r/2)). The streaming score is the complement. The hybrid advice output returns 1 if r < 50 (meaning batch is likely enough) and 2 otherwise (meaning you likely need streaming or hybrid). This is a heuristic, not a law – it helps you articulate your requirements.

Worked Example

Using the default values: batch interval = 24 hours, streaming latency target = 5 seconds, data volume = 100 GB/day, real-time query ratio = 20%.

1. Compute s/60 = 5/60 = 0.083.
2. Compute r/2 = 20/2 = 10.
3. Sum = 10.083.
4. Batch score = 100 - min(100, 10.083) = 89.9.
5. Streaming score = min(100, 10.083) = 10.1.
6. Since r = 20 (less than 50), hybrid advice = 1 (batch is likely sufficient).

The calculator suggests that with a 20% real-time query ratio and a 5-second latency target, batch is still a strong fit because most of your workload tolerates delays.

Background: Why This Choice Matters

Data pipelines move data from source systems (databases, logs, IoT sensors, APIs) to destinations like data warehouses, data lakes, or machine learning models. The architecture determines how fresh the data is, how much infrastructure you need, and how complex your code becomes. Batch processing has existed for decades – it collects data over a period (hourly, daily) and processes it all at once. Streaming processes each event as it arrives, often within milliseconds. Hybrid architectures combine both, using streaming for time-sensitive paths and batch for heavy historical computation.

The choice is not permanent. Many organizations start with batch and add streaming later. The key is to understand the latency requirement of each use case, not to pick one architecture for the entire company.

How Batch Pipelines Work

Batch pipelines follow a simple cycle: extract, transform, load (ETL) or extract, load, transform (ELT).

• Scheduling – a job runs at fixed intervals (e.g., every night at 2 AM) using tools like cron, Apache Airflow, or a cloud scheduler.
• Extraction – pull data from sources, often as full snapshots or incremental changes since the last run.
• Transformation – clean, join, aggregate, and format the data using SQL or a framework like Spark or dbt.
• Loading – write the result to a warehouse or data lake.

Batch is predictable: you know exactly when data will be ready, and failures can be retried without losing real-time events. It is also cost-efficient for large volumes because you can use spot instances or reserved capacity that runs for a few hours.

How Streaming Pipelines Work

Streaming pipelines process unbounded data continuously. The core components are:

• Event ingestion – a message broker like Apache Kafka or AWS Kinesis captures events as they happen.
• Stream processing – a framework such as Apache Flink, Spark Streaming, or Kafka Streams applies transformations in small micro-batches or event-by-event.
• Sinks – results are written to real-time dashboards, alerts, or a data store.

Streaming excels at fraud detection, live inventory, and monitoring. It also enables “event-driven� architectures where downstream systems react immediately. However, it demands more operational care: you must handle out-of-order events, state management, and exactly-once semantics, which are harder than batch’s simple retry model.

Hybrid: Lambda and Kappa Architectures

Hybrid architectures try to get the best of both. The two classic patterns are:

• Lambda architecture – runs a batch layer for accurate, comprehensive results and a speed layer for low-latency approximations. A serving layer merges both. It is powerful but doubles your codebase because you implement the same logic twice.
• Kappa architecture – treats everything as a stream. Batch is simulated by replaying historical events through the streaming engine. This simplifies code but requires your streaming system to handle large replays efficiently.

In practice, many teams use a simpler hybrid: stream data into a lake or warehouse in near-real-time, then run batch jobs on that fresh data for complex analytics. This avoids maintaining two separate processing engines while still giving sub-hour freshness.

How to Choose: Key Factors

Ask yourself these questions, in order:

1. What is the business SLA for data freshness? If a decision can wait until tomorrow, batch is fine. If a customer sees “out of stock� when the item is actually available, you need streaming.
2. What is your data volume and event rate? Batch handles high volume with lower cost per record. Streaming can handle high throughput but costs more per event due to continuous compute.
3. What is your team’s skill set? Streaming requires expertise in distributed systems, state management, and backpressure. Batch is easier to debug and test.
4. What is your budget for infrastructure and engineering time? Streaming infrastructure (brokers, processing clusters) runs 24/7. Batch only runs when scheduled.
5. Do you need exactly-once processing? Both can achieve it, but streaming’s exactly-once is more complex and often slower.

A common misconception is that streaming is always “better� because it is faster. In reality, faster is only valuable if someone acts on it. If your dashboards are checked daily, streaming just adds cost and complexity.

Common Problems and Misconceptions

• “Streaming is always low-latencyâ€� – Micro-batch streaming (e.g., Spark Streaming with 10-second intervals) adds that delay. True event-at-a-time processing (Flink, Kafka Streams) is lower but more complex.
• “Batch is outdatedâ€� – Batch remains the backbone of most data platforms because it is reliable, cheap, and easy to audit. Many “real-timeâ€� systems actually use micro-batches.
• “You must pick oneâ€� – Most large companies run both. A common pattern is streaming for ingestion and batch for heavy transformation.
• Failure handling – Batch can simply re-run. Streaming needs checkpoints and idempotent sinks to avoid duplicates or gaps.
• Schema evolution – Streaming makes schema changes harder because old events may still be in flight. Batch can easily reprocess historical data with a new schema.

Cost and Operational Trade-offs

Batch typically costs less in raw compute because you can run on discounted instances and shut down between runs. Storage is cheap; compute is the main cost. Streaming requires always-on brokers and processors, which increases base cost even when traffic is low. However, streaming can reduce cost elsewhere: for example, avoiding expensive database lookups by keeping a real-time aggregation in memory.

Operationally, batch is simpler: if a job fails, you see an alert and re-run. Streaming requires monitoring consumer lag, state sizes, and rebalancing. You also need to plan for data replay – if you discover a bug, can you reprocess the last week? Batch can do this easily; streaming may need to re-ingest from source or keep long-term state.

Frequently Asked Questions

Can I start with batch and add streaming later? Yes. Many architectures begin with batch, then add a streaming path for specific use cases like alerts or real-time dashboards. Keep your ingestion layer decoupled (e.g., write to a lake in both batch and streaming modes) so you can evolve without rewriting everything.

Is micro-batch considered streaming? Micro-batch (e.g., Spark Streaming with 5-second intervals) is a hybrid: it processes small chunks, so latency is seconds, not milliseconds. It is easier to implement than true event-at-a-time streaming and is often a good compromise.

What is the simplest architecture for a small team? Start with batch using a managed scheduler (e.g., AWS Glue, Airflow) and a cloud warehouse. Only add streaming when you have a concrete requirement that batch cannot meet. Avoid building a Lambda architecture from scratch unless you have dedicated data engineers.

How do I handle exactly-once semantics in streaming? Use a processing engine that supports it (Flink, Kafka Streams) and make your sinks idempotent – meaning writing the same event twice produces the same result. This usually involves using a unique key or transaction. It adds overhead, so only enable it where duplicates are unacceptable.

Does streaming require a message broker like Kafka? Not necessarily. You can use simpler tools like AWS Kinesis, Google Pub/Sub, or even a database log-based CDC (change data capture). The broker choice depends on your ecosystem and throughput needs.

What is CDC and how does it relate to pipelines? Change data capture reads database transaction logs to capture inserts, updates, and deletes in real time. Tools like Debezium can feed CDC events into a streaming pipeline, enabling real-time replication without double-writing to the database.

Related

• Second Wind Quest Walkthrough: Where Winds Meet
• Enshrouded Weapon Upgrade Guide: How to Improve Gear
• Connect Phone to Controller for Cloud Gaming

in t | Batch interval (hours) | h | 24 | 0 | 8760
in s | Streaming latency target (seconds) | s | 5 | 0 | 3600
in d | Data volume per day (GB) | GB | 100 | 0 | 100000
in r | Real-time query ratio (%) | % | 20 | 0 | 100
out batch_score | Batch suitability score | 0-100 | 100 - min(100, (s / 60) + (r / 2)) | 0
out stream_score | Streaming suitability score | 0-100 | min(100, (s / 60) + (r / 2)) | 0
out hybrid_advice | Recommended approach | - | 0 * sqrt(max(0, s - 60)) + (r < 50 ? 1 : 2) | 0