Skip to content

Add default-structure tariff generation from monthly_rates YAML #372

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alxsmith merged 48 commits into from
Mar 25, 2026
Merged

Add default-structure tariff generation from monthly_rates YAML #372

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
ca96e3f
Add default-structure tariff generation from monthly_rates YAML
alxsmith Mar 19, 2026
4199bd5
Add Justfile recipes and parameterize workflow for base tariff pattern
alxsmith Mar 19, 2026
301ca37
Configure RI to use default-structure tariffs
alxsmith Mar 19, 2026
1d35a2d
Update RI tariffs from default-structure CAIRO runs
alxsmith Mar 19, 2026
8494cc8
Add default-structure tariff JSONs for all NY utilities
alxsmith Mar 20, 2026
fc1502e
Configure NY to use default-structure tariffs and ResStock loads
alxsmith Mar 20, 2026
3574ecc
Add HP seasonal discount plans to context/plans/
alxsmith Mar 20, 2026
3a9bbec
feat(seasonal): derive HP seasonal discount from base URDB
alxsmith Mar 23, 2026
10da096
fix(validate): skip non-HP keys in seasonal winter/summer ordering check
alxsmith Mar 23, 2026
e68b876
docs: Plan B seasonal discount and orchestration
alxsmith Mar 23, 2026
1f6a840
chore(ny): add trailing newlines to default electric tariff JSON
alxsmith Mar 23, 2026
97606e6
chore(ri): refresh RIE tariffs and subclass RR after latest run
alxsmith Mar 23, 2026
1a5d5f6
linter
alxsmith Mar 23, 2026
bb0647d
fix(seasonal): derive HP seasonal rates from annual bills using blend…
sherryzuo Mar 23, 2026
527ccce
fix(loads): use grid_cons = max(total - abs(pv), 0) to match CAIRO bi…
sherryzuo Mar 23, 2026
3cac3e2
add rest of the tariffs and done
sherryzuo Mar 23, 2026
770ff0c
fix(tariffs): include flat supply riders in TOU slot rates
alxsmith Mar 23, 2026
92c95cc
chore(ny): regenerate electric tariffs after all-pre with resstock loads
alxsmith Mar 23, 2026
3e5a3fe
chore(ny): recompute revenue requirements using resstock monthly loads
alxsmith Mar 23, 2026
9673b7e
chore(ny): switch scenario YAMLs from flat to default-structure tariffs
alxsmith Mar 23, 2026
0b47ef9
chore(ny): add nonhp_default tariffs for all 7 utilities
alxsmith Mar 23, 2026
eac0ba2
chore(ri): add trailing newlines to calibrated tariff JSONs
alxsmith Mar 23, 2026
bd021a9
fix(pre): Correct seasonal-TOU default tariff generation for supply
alxsmith Mar 24, 2026
baee74d
config(ny): Point scenarios and rev requirement at default-base tariffs
alxsmith Mar 24, 2026
109d3e1
config(ny): Regenerate electric URDB JSONs for default structure
alxsmith Mar 24, 2026
6d7745e
config(ny): Add electric tariff maps for default-base comparisons
alxsmith Mar 24, 2026
813182f
Fix EC2 EBS volume proliferation on dev-setup
jpvelez Mar 24, 2026
8176917
Add runs 17-20 to scenario_rie and corresponding just task stubs
jpvelez Mar 24, 2026
f711116
Add corrected rie_nonhp_flat
jpvelez Mar 24, 2026
317be06
Add new hp_flat tariff maps
jpvelez Mar 24, 2026
82c7303
Finish hp_flat implementation for ri
jpvelez Mar 24, 2026
6dff1be
Slightly new rie_hp_seasonalTOU and supply
jpvelez Mar 24, 2026
a04e727
Correct newlines so rerunning all-pre doesn't create false positives …
jpvelez Mar 24, 2026
a6d2567
Fix fixed charges in RIE tariffs
jpvelez Mar 24, 2026
58e77aa
Fix utility_assigment path and add new runs to master bill and bat
jpvelez Mar 25, 2026
4d3d9e1
Updated tariffs etc from batch ri_20260324_r1-20_fixedcharges_hpflat
jpvelez Mar 25, 2026
d1fa41c
linter
alxsmith Mar 25, 2026
316f5c3
Merge remote-tracking branch 'origin/main' into 371-add-default-struc…
alxsmith Mar 25, 2026
9914049
Disable runs 17-20 in sequential and parallel-tracks orchestration
alxsmith Mar 25, 2026
01d173b
Add tariff generation pipeline documentation
alxsmith Mar 25, 2026
3fdd160
Update NY config artifacts from ny_20260325b batch runs
alxsmith Mar 25, 2026
93013a9
Vectorize solar net-metering in _vectorized_run_system_revenues
alxsmith Mar 25, 2026
19c76e0
Fix formatting from dprint/ruff auto-fixes
alxsmith Mar 25, 2026
7b7232a
Update calibrated tariffs from ny_20260325b ConEd + RI runs
alxsmith Mar 25, 2026
9de89c9
Fix trailing newlines in pre-calibrated NY tariff JSONs
alxsmith Mar 25, 2026
8f15555
Add ConEd demand-flex runs 13-16 to post-processing valid pairs
alxsmith Mar 25, 2026
ffa7733
Update calibrated ConEd tariffs from ny_20260325b batch run
alxsmith Mar 25, 2026
2ae39e8
Update CenHud revenue requirement test expectations to match updated …
alxsmith Mar 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
6 changes: 6 additions & 0 deletions Justfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,12 @@ up-aws-list:
_terraform: aws
bash infra/install-terraform.sh

# Create the persistent EBS data volume (run once, ever). Safe to run again —

# it's a no-op if a volume tagged rate-design-platform-data already exists.
create-volume: _terraform
bash infra/create-volume.sh

# Set up EC2 instance (run once by admin)

# Idempotent: safe to run multiple times
Expand Down
39 changes: 20 additions & 19 deletions context/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,25 +110,26 @@ How CAIRO works: LMI, demand flex, tax rate, tiered rates, performance, parallel

ResStock, EIA, NYISO, ISO-NE: where data comes from, how to read and prepare it.

| File | Purpose |
| -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| resstock_data_preparation_run_order.md | Run order for ResStock data: fetch → identify HP/heating/natgas → utility assignment (standard) → copy to sb release (incl. metadata_utility) → approximate non-HP load → sync S3 sb release to EBS |
| resstock_lmi_metadata_guide.md | ResStock 2024.2 parquet metadata: columns for LMI tier assignment, FPL/SMI, income |
| parquet_reads_local_vs_s3.md | Reading Parquet from local disk vs S3: per-GET overhead, file discovery, Hive filters vs path construction, best practices for whole-state and per-utility reads |
| polars_laziness_and_validation.md | Polars LazyFrame best practices: when to collect, runtime data-quality asserts vs laziness/streaming, strategies for small and large data |
| nyiso_lbmp_zonal_data_sources.md | NYISO Day-Ahead/Real-Time zonal LBMP: MIS ZIP vs gridstatus vs NYISOToolkit; data samples |
| eia_861_supply_vs_delivery_stats.md | EIA-861 supply vs delivery stats: definitions and use |
| compare_resstock_eia861_loads.md | ResStock hourly load vs EIA-861 residential sales comparison: script usage, defaults, interpretation |
| investigate_resstock_eia_load_discrepancy.md | ResStock vs EIA load discrepancy by utility: MF non-HVAC adjustment, column-by-column SF/MF ratios, workflow, design decisions, and relation to compare_resstock_eia861_loads |
| isone_fca_yaml_data_guide.md | ISO-NE FCA assumptions YAML data dictionary: what each field means (clearing price, CSO, months), where to find source data online (FCA results report, CELT Report Table 4.1), and how to update for a new year |
| ny_utility_assignment_resstock.md | NY ResStock utility assignment: small gas utilities (zero prior), renormalize, nearest-neighbor donor for PUMAs with no gas left, PUMA id normalization, Justfile/data flow, tests |
| approximate_non_hp_load.md | MF highrise non-HP load approximation: nearest-neighbor RMSE, replace HVAC columns, update/sink, validation |
| ny_lmi_discounts_genability_encoding.md | EAP/LMI encoding in NY Genability electric tariffs: which utilities encode EAP, tier/heating structure, amount comparison to lmi_discounts_in_ny.md |
| tariff_rates_and_genability.md | Tariff/rate/rider/charge terminology, regulatory underpinnings (rate cases, rider proceedings, tariff books, leaf/revision versioning), Genability/Arcadia data model and APIs, and how to reconstruct historical monthly charges (API sequence, date-boundary subtleties, validation via Calculate API) |
| ny_residential_charges_in_bat.md | All NY residential electric charges (all 7 utilities) evaluated for BAT/bill calc: charge type taxonomy, master table, charge-by-charge analysis; Genability data quirks; discover → classify workflow for charge_decisions. |
| ri_residential_charges_in_bat.md | All RI residential electric charges (RIE A-16) evaluated for BAT/bill calc: charge type taxonomy, summary table, charge-by-charge analysis (base delivery, cost recon, program surcharges, sunk-cost, LMI recovery, LRS/ISO-NE supply decomposition, RES, GET). |
| ny_genability_charge_fetch_map.md | Exhaustive charge-level table for NY Genability tariffs: tariffRateId, fetch_type, variableRateKey, master_charge, decision; used to fetch 2025 monthly rates from Arcadia API for top-up implementation. |
| psegli_revenue_requirement_estimation.md | PSEG-LI revenue requirement estimation: why no rate case, TOU overcount problem, bill-proportional method from LIPA budget, charge classification, implementation via `estimate_psegli_rr.py`. |
| File | Purpose |
| -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| resstock_data_preparation_run_order.md | Run order for ResStock data: fetch → identify HP/heating/natgas → utility assignment (standard) → copy to sb release (incl. metadata_utility) → approximate non-HP load → sync S3 sb release to EBS |
| resstock_lmi_metadata_guide.md | ResStock 2024.2 parquet metadata: columns for LMI tier assignment, FPL/SMI, income |
| parquet_reads_local_vs_s3.md | Reading Parquet from local disk vs S3: per-GET overhead, file discovery, Hive filters vs path construction, best practices for whole-state and per-utility reads |
| polars_laziness_and_validation.md | Polars LazyFrame best practices: when to collect, runtime data-quality asserts vs laziness/streaming, strategies for small and large data |
| nyiso_lbmp_zonal_data_sources.md | NYISO Day-Ahead/Real-Time zonal LBMP: MIS ZIP vs gridstatus vs NYISOToolkit; data samples |
| eia_861_supply_vs_delivery_stats.md | EIA-861 supply vs delivery stats: definitions and use |
| compare_resstock_eia861_loads.md | ResStock hourly load vs EIA-861 residential sales comparison: script usage, defaults, interpretation |
| investigate_resstock_eia_load_discrepancy.md | ResStock vs EIA load discrepancy by utility: MF non-HVAC adjustment, column-by-column SF/MF ratios, workflow, design decisions, and relation to compare_resstock_eia861_loads |
| isone_fca_yaml_data_guide.md | ISO-NE FCA assumptions YAML data dictionary: what each field means (clearing price, CSO, months), where to find source data online (FCA results report, CELT Report Table 4.1), and how to update for a new year |
| ny_utility_assignment_resstock.md | NY ResStock utility assignment: small gas utilities (zero prior), renormalize, nearest-neighbor donor for PUMAs with no gas left, PUMA id normalization, Justfile/data flow, tests |
| approximate_non_hp_load.md | MF highrise non-HP load approximation: nearest-neighbor RMSE, replace HVAC columns, update/sink, validation |
| ny_lmi_discounts_genability_encoding.md | EAP/LMI encoding in NY Genability electric tariffs: which utilities encode EAP, tier/heating structure, amount comparison to lmi_discounts_in_ny.md |
| tariff_rates_and_genability.md | Tariff/rate/rider/charge terminology, regulatory underpinnings (rate cases, rider proceedings, tariff books, leaf/revision versioning), Genability/Arcadia data model and APIs, and how to reconstruct historical monthly charges (API sequence, date-boundary subtleties, validation via Calculate API) |
| ny_residential_charges_in_bat.md | All NY residential electric charges (all 7 utilities) evaluated for BAT/bill calc: charge type taxonomy, master table, charge-by-charge analysis; Genability data quirks; discover → classify workflow for charge_decisions. |
| ri_residential_charges_in_bat.md | All RI residential electric charges (RIE A-16) evaluated for BAT/bill calc: charge type taxonomy, summary table, charge-by-charge analysis (base delivery, cost recon, program surcharges, sunk-cost, LMI recovery, LRS/ISO-NE supply decomposition, RES, GET). |
| ny_genability_charge_fetch_map.md | Exhaustive charge-level table for NY Genability tariffs: tariffRateId, fetch_type, variableRateKey, master_charge, decision; used to fetch 2025 monthly rates from Arcadia API for top-up implementation. |
| psegli_revenue_requirement_estimation.md | PSEG-LI revenue requirement estimation: why no rate case, TOU overcount problem, bill-proportional method from LIPA budget, charge classification, implementation via `estimate_psegli_rr.py`. |
| tariff_generation_pipeline.md | Tariff generation pipeline: Genability → monthly_rates YAML → flat and default-structure URDB v7 JSONs. Covers fetch_monthly_rates.py, charge classification, monthly_rates YAML structure, create_flat_tariffs.py (top-down from RR), create_default_structure_tariffs.py (bottom-up from filed rates), BASE_TARIFF_PATTERN, Justfile wiring (all-pre), CAIRO calibration, and why flat vs default rates differ. |

### code/marginal_costs/

Expand Down
Loading