Spark MMS

Custom Electricity Market Management System (MMS) CSV reader library for Apache Spark. This library can be used to efficiently read MMS data model reports in bulk. It uses Spark's DataSource V2 API and reads files in AEMO's MMS Data Model CSV format.
Features

• Partitions large files to avoid out of memory (OOM) errors
• Supports multiple reports per file
• Supports zipped files
• Supports filter pushdown
• Supports column pruning
• Reads report schemas from input files
• Registers {report_type, report_subtype, report_version} as temporary tables
• Supports quoted string fields

Change Log

• [0.4.0] Bump Spark version to 3.5.1

Spark MMS

Custom Electricity Market Management System (MMS) CSV reader library for Apache Spark. This library can be used to efficiently read MMS data model reports in bulk. It uses Spark's DataSource V2 API and reads files in AEMO's MMS Data Model CSV format.

Features

• Partitions large files to avoid out of memory (OOM) errors
• Supports multiple reports per file
• Supports zipped files
• Supports filter pushdown
• Supports column pruning
• Reads report schemas from input files
• Registers {report_type, report_subtype, report_version} as temporary tables
• Supports quoted string fields