Skip to content

Latest commit

 

History

History
31 lines (26 loc) · 1.79 KB

Batch_Stream_Processing.md

File metadata and controls

31 lines (26 loc) · 1.79 KB

Batch And Stream Processing

Comparison

Batch processing Stream processing
Responsiveness No response to the client needed No response to the client needed
Timelineness On demand Real-time
Input Bounded input with finite size. A large amount of data Input has no boundary (infinite streams)
Output Materialized views, aggregated metrics, etc. Materialized views, aggregated metrics, etc.
Performance measurement Throughput Throughput, latency
Fault tolerance Better (a fixed set of input data) More challenging (the input data keeps flowing in)
Example MapReduce
  • Apache Storm
  • Apache Flink
  • Apache Spark
  • Diagram Screenshot 2024-01-03 at 11 13 55 AM Screenshot 2024-01-03 at 11 14 05 AM

    Architecture patterns

    Lambda

    • Concepts

      • A system that contains two processing paths (batch and streaming) simultaneously.
    • Cons

      • There are two codebases to maintain.
    • Diagram

      figure-10-lambda-and-kappa-architectures-QONSAGXY

    Kappa

    • Concepts

      • A system that combines the batch and streaming in one processing path.
      • Handle both real-time data processing and continuous data reprocessing using a single stream processing engine.
    • Diagram

      figure-10-lambda-and-kappa-architectures-QONSAGXY