module5-batch-processing

Batch processing with Spark

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 4.0.1
      /_/

Using Python version 3.14.2 (main, Jan 14 2026 23:37:46)
Spark context Web UI available at http://192.168.15.5:4043
Spark context available as 'sc' (master = local[*], app id = local-1769554629678).
SparkSession available as 'spark'.
Cmd click to launch VS Code Native REPL

Getting Started

1. Spin up the Spark Cluster

docker compose up -d

2. Refer to the specific implementations for docs on how to run the pipeline:

• Kotlin
• Scala
• PySpark