Skip to content

Commit

Permalink
Merge pull request #238 from SANDAG/docs_update_aku
Browse files Browse the repository at this point in the history
update reporting docs
  • Loading branch information
anneku authored Nov 12, 2024
2 parents c986aa6 + 5f3f497 commit 9e76c26
Showing 1 changed file with 20 additions and 1 deletion.
21 changes: 20 additions & 1 deletion docs/design/report/report.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,22 @@
# Reporting Framework

Details of reporting components.
**Reporting Process Overview:**

1. **ABM3 model output files are stored to data lake:**
- Model outputs are written to the data lake immediately following ABM3 model run completion.
- Output CSV files are converted to Parquet format before writing to the data lake.
- Each model run is assigned a unique scenario ID.

2. **Data lake files are loaded to Delta tables:**
- Each output file in the data lake is loaded into its corresponding Delta table. For example, the trips output file is loaded into the trips Delta table, the persons output file is loaded into the persons Delta table, etc.
- Delta tables store the results from all model runs, organized by scenario ID.

3. **Delta Tables are processed in Databricks:**
- Delta tables are read, transformed, and aggregated as needed to support analysis and reporting requirements.
- Once transformations are complete, the resulting data is written back to the data lake as new Delta tables or used to update existing tables.
- These new Delta tables are also organized by scenario ID, making it easier to manage and query specific versions of processed data.

4. **Delta tables are ingested by Power BI:**
- Power BI reads the data from the Delta tables.
- Power BI report templates with various metrics of interest are automatically refreshed with new model run outputs.
- Metrics can easily be compared across different scenario IDs.

0 comments on commit 9e76c26

Please sign in to comment.