Batch vs stream processing
Model answer
Batch processing collects data over a period and processes it in one run on a bounded dataset, for example a nightly job. It is simple, efficient for large volumes and easy to retry. Latency is minutes to hours. Typical tools are Spark and warehouse SQL jobs.
Stream processing handles events continuously as they arrive, on an unbounded dataset, with latency from milliseconds to seconds. Typical tools are Kafka with Flink, Spark Structured Streaming or Kafka Streams. It suits fraud detection, live dashboards and alerts.
Streaming adds harder problems: event time versus processing time, late and out-of-order events handled with watermarks and windows, state management, and delivery guarantees (at-least-once with idempotent writes, or exactly-once semantics where supported).
Choose by the latency the business actually needs. Many cases are served well by frequent micro-batches. Some systems combine both, running batch for history and streaming for fresh data. Do not pay the extra complexity of streaming when daily freshness is enough.
Key points
- Batch: bounded data, higher latency, simple and efficient
- Stream: unbounded events, low latency, continuous processing
- Streaming needs event time, watermarks, windows and state handling
- Choose by the latency the business needs
Likely follow-ups
- What is a watermark and why do you need one?
- How do you handle duplicates in an at-least-once stream?
Common mistakes
- Choosing streaming when daily freshness would do
- Using processing time when the business needs event time