Conversation
…nalytical-validation-framework-and-elasticity-sensitivity-analysis
Implements utils/post/diagnose_demand_flex.py: an analytical diagnostic that loads real HP building loads and TOU derivation data to sweep candidate elasticity values (-0.02 to -0.20), compute peak reduction % and rate-arbitrage savings per HP building, and recommend the elasticity that best matches the Arcturus 2.0 "no enabling technology" empirical target for each NY utility. Runs in ~15s per utility without requiring a CAIRO run. Made-with: Cursor
Implements utils/post/validate_demand_flex_shift.py: reproduces the constant-elasticity demand-response shift analytically using the same functions CAIRO uses, then verifies correctness via three checks: energy conservation (per-building kWh preserved), direction (peak kWh decreases, off-peak increases), and cross-check against CAIRO's saved demand_flex_elasticity_tracker.csv. All 7 NY utilities pass at ε=-0.10. Produces five diagnostic plots per utility: aggregate daily load profile (pre/post with shaded fill), net shift by hour bars, month×hour heatmap, per-building peak reduction distribution, and per-building daily profiles at representative consumption percentiles. Also writes CSV/parquet data outputs for report reuse. Made-with: Cursor
context/methods/tou_and_rates/demand_flex_elasticity_calibration.md: accessible writeup of the per-utility elasticity calibration methodology, Arcturus 2.0 empirical anchor, two savings mechanisms (rate arbitrage vs RR reduction), results table (ε=-0.10 or -0.12 per utility), and known limitations. context/plans/demand-flex_elasticity_calibration.md: living plan document for the broader demand-flex calibration and validation work. Made-with: Cursor
rate_design/hp_rates/ny/Justfile: add validate-demand-flex, validate-demand-flex-all, and diagnose-demand-flex recipes for running the shift validation and elasticity sweep diagnostic from the command line. context/README.md: index the new demand_flex_elasticity_calibration.md context document. Made-with: Cursor
5 tasks
context/methods/tou_and_rates/demand_flex_elasticity_calibration.md: - Add Validation section: all-utilities pass results at ε=-0.10 (energy conservation, direction, CAIRO tracker cross-check) - Add per-utility peak reduction table at ε=-0.10 - Document the five diagnostic plot types and what they show - Add invoke instructions for the new Justfile recipes context/plans/demand-flex_elasticity_calibration.md: - Update todos to reflect Phase 1 completion (diagnostic script, validation script, Justfile recipes, results review all done) - Add status table at the top of the plan body - Add pending items: scenario YAML update, optional Phase 2 sweep Made-with: Cursor
utils/cairo.py and utils/demand_flex.py now accept a season-keyed dict for the elasticity parameter in addition to a scalar. The shifting pipeline resolves the per-season value inside _shift_season so summer and winter can have independently calibrated epsilons without any structural change to the call sites. Made-with: Cursor
run_scenario.py now passes the elasticity value from the scenario YAML directly to the CAIRO runtime unchanged (dict or scalar), enabling seasonal elasticity to flow from config/periods/*.yaml through create_scenario_yamls into the actual CAIRO run. Made-with: Cursor
Moves the elasticity calibration script from utils/post to utils/pre
since its primary output is config/periods/*.yaml, which is consumed
before a CAIRO run. Changes from the original:
- Dual Arcturus model: computes both no-tech (slope=-0.065) and
with-tech (slope=-0.111, corrected interaction term) recommendations
- Seasonal elasticities: matches each season's price ratio against its
own Arcturus target independently
- --write-periods: writes both elasticity and elasticity_with_tech dicts
to config/periods/{utility}.yaml
- --compare-batch / --with-tech: compares analytical savings against a
completed CAIRO batch, selecting the right reference set
- --epsilon-start/--epsilon-end/--epsilon-step: replaces hardcoded list
with a configurable sweep range (default -0.04 to -0.50, step -0.02)
- ty/ruff fixes: cast for period_rate, .to_series().dt.month.to_numpy()
for DatetimeIndex.month, removed dead assignments
Made-with: Cursor
…ing_tech _row_to_run now reads enabling_tech from the Google Sheet row and selects elasticity_with_tech from the periods YAML when enabling_tech is empty, true, yes, or 1 (with-tech is the default). Only explicit false/no/0 selects the no-tech elasticity key. This makes enabling technology opt-out rather than opt-in, matching the intended workflow. Made-with: Cursor
json.dumps() does not append a newline, causing every _calibrated.json to fail the end-of-file-fixer pre-commit hook and show spurious diffs. Made-with: Cursor
- Add comprehensive usage examples (just recipes and direct uv run) to the module docstring covering scalar, seasonal, no-tech, with-tech, and CAIRO cross-check invocations - Fix DatetimeIndex.month access: use .to_series().dt.month.to_numpy() to avoid ty unresolved-attribute and pandas alignment errors - Cast period_rate to pd.Series to satisfy ty invalid-assignment check - Remove unused bldg_level assignment and dead p_flat groupby block - Remove stale type: ignore[arg-type] comments - Fix set_xticklabels: pass list[str] instead of range Made-with: Cursor
diagnose_demand_flex.py, sensitivity_demand_flex.py, and validate_demand_flex.py are replaced by calibrate_demand_flex_elasticity (now in utils/pre) and validate_demand_flex_shift. The corresponding test file is also removed as it tested the deleted validate_demand_flex. Made-with: Cursor
- Move module reference from utils.post to utils.pre - Add comprehensive usage examples for all invocation patterns (default sweep, utility subset, custom range, --write-periods, --compare-batch with and without --with-tech) Made-with: Cursor
Full calibration sweep (epsilon -0.04 to -0.50, step -0.02) across all
7 NY utilities. Each periods YAML now has:
elasticity: {summer: ..., winter: ...} # Arcturus no-tech match
elasticity_with_tech: {summer: ..., winter: ...} # Arcturus with-tech match
No-tech summer epsilons range from -0.10 (NiMo/NYSEG/RGE) to -0.14
(CenHud/OR/PSEG-LI). With-tech values are ~1.5-1.7x larger. Seasonal
elasticities independently match each season's price ratio against the
Arcturus target rather than using a single annual approximation.
Made-with: Cursor
Scenario YAMLs regenerated from the Google Sheet via create_scenario_yamls. Demand-flex runs now carry a seasonal elasticity dict read from the utility's periods YAML (elasticity_with_tech when enabling_tech is empty or true, elasticity otherwise). This replaces the previous scalar elasticity value and enables summer/winter-specific load shifting in CAIRO. Made-with: Cursor
… run Promoted calibrated tariff JSONs from the latest CenHud CAIRO run. All files now end with a trailing newline (fixed in copy_calibrated_tariff_from_run). Made-with: Cursor
… full results - demand_flex_elasticity_calibration.md: corrected Arcturus with-tech coefficients (interaction term: slope=-0.065 + -0.046 = -0.111), updated results tables with full 7-utility seasonal sweep for both no-tech and with-tech, refreshed invocation examples, added note on enabling_tech defaulting to with-tech - context/README.md: index updated to reflect calibration script move to utils/pre Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the demand-flex elasticity calibration and validation framework for NY HP rate design. The core analytical question: what elasticity parameter makes our constant-elasticity load-shift model reproduce the aggregate demand response observed in real-world TOU pricing pilots?
What this PR adds
Calibration (
utils/pre/calibrate_demand_flex_elasticity.py) — Sweeps candidate elasticity values for each NY utility and finds the one whose peak reduction % matches the Arcturus 2.0 empirical target. Computes both "no enabling technology" and "with enabling technology" recommendations using the corrected Arcturus regression (single model with interaction term: no-tech slope = -0.065, with-tech slope = -0.111 = -0.065 + -0.046 interaction). Writes both sets of seasonal elasticities toconfig/periods/{utility}.yamlunderelasticityandelasticity_with_tech. Sweep range is configurable via--epsilon-start/--epsilon-end/--epsilon-step(default -0.04 to -0.50, step -0.02). Can compare analytical savings estimates against a completed CAIRO batch (--compare-batch,--with-tech).Seasonal elasticity runtime (
utils/cairo.py,utils/demand_flex.py,run_scenario.py) — The shifting pipeline now accepts a{season: epsilon}dict in addition to a scalar, resolving the per-season value inside_shift_season. This lets summer and winter have independently calibrated epsilons matching their respective price ratios.Scenario YAML automation (
utils/pre/create_scenario_yamls.py) — Readsenabling_techfrom the Google Sheet and selectselasticity_with_techfrom the periods YAML when empty or true (default),elasticityonly when explicitly false/no/0. Scenario YAMLs for all 7 NY utilities regenerated with seasonal elasticity dicts.Validation (
utils/post/validate_demand_flex_shift.py) — Reproduces the shift analytically and checks: (1) energy conservation, (2) correct direction, (3) match against CAIRO'sdemand_flex_elasticity_tracker.csv. All 7 NY utilities pass at ε=-0.10. Produces 5 diagnostic plots per utility.Calibrated results — Full sweep across all 7 NY utilities. Periods YAMLs updated with calibrated values. No-tech summer epsilons: -0.10 (3-hr peak utilities) to -0.14 (5-hr peak utilities). With-tech values are ~1.5-1.7x larger. CenHud calibrated tariffs updated from the
ny_20260326_elast_seasonal_techrun.Docs (
context/methods/tou_and_rates/demand_flex_elasticity_calibration.md) — Methodology, full 7-utility results tables for both no-tech and with-tech, corrected Arcturus model description, complete invocation examples.Reviewer focus
enabling_techdefaulting to with-tech: an emptyenabling_techcell in the Google Sheet means with-tech (opt-out rather than opt-in). Only explicitfalse/no/0selects no-tech. This matches the intended workflow where enabling technology is assumed unless overridden.Closes #377