diff --git a/README.md b/README.md index f701a21..cd53096 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ uvx --env-file=.env --from git+https://github.com/switchbox-data/tariff_fetch ta Or, for gas tariffs: ```bash -uvx --env-file=.env --from git+https://github.com/switchbox-data/tariff_fetch tariff-fetch-gas +uvx --env-file=.env --from git+https://github.com/switchbox-data/tariff_fetch tariff-fetch gas ``` ## Installation @@ -75,43 +75,61 @@ pip install -e . ```bash python -m tariff_fetch.cli [OPTIONS] -python -m tariff_fetch.cli_gas [OPTIONS] ``` With uv: ```bash uv run tariff-fetch [OPTIONS] -uv run tariff-fetch-gas [OPTIONS] -uv run tariff-fetch-arcadia-urdb MASTER_TARIFF_ID YEAR [OPTIONS] +uv run tariff-fetch ni arcadia MASTER_TARIFF_ID [EFFECTIVE_DATE] [OPTIONS] +uv run tariff-fetch ni rateacuity fuzzy STATE UTILITY_QUERY --tariff TARIFF_QUERY [--tariff TARIFF_QUERY ...] [OPTIONS] +uv run tariff-fetch ni rateacuity eia-id EIA_ID --tariff TARIFF_QUERY [--tariff TARIFF_QUERY ...] [OPTIONS] +uv run tariff-fetch gas [OPTIONS] +uv run tariff-fetch gas ni STATE UTILITY_QUERY --tariff TARIFF_QUERY [OPTIONS] +uv run tariff-fetch gas urdb [OPTIONS] +uv run tariff-fetch gas urdb ni STATE UTILITY_QUERY --year YEAR --tariff TARIFF_QUERY [OPTIONS] +uv run tariff-fetch urdb ni MASTER_TARIFF_ID YEAR [OPTIONS] ``` With Just: ```bash just cli -just cligas ``` Options: - `--state` / `-s`: two-letter state abbreviation (default: prompt) -- `--providers` / `-p`: (only for electricity benchmarks) repeat per provider (`genability`, `openei`, `rateacuity`) +- `--provider` / `-p`: provider to fetch (`genability`, `openei`, `rateacuity`) - `--output-folder` / `-o`: directory for exports (default: `./outputs`) +- `--effective-date`: provider query date in `YYYY-MM-DD` format +- `--no-input`: fail instead of prompting for interactive input +- `--log-dir`: directory for log files +- `--log-file`: exact log file path Omitted options will trigger interactive prompts. +Use `--no-input` for automation or CI runs when you want missing required inputs to fail fast instead of opening an +interactive prompt. + +When the CLI reaches the utility selection step, it caches the EIA utility parquet for 1 hour in the platform-specific +user cache directory so repeated runs do not re-download it every time. You can clear that cache with: + +```bash +uv run tariff-fetch cache clear +``` + ### Examples ```bash # Fully interactive run uv run tariff-fetch -# Scripted run for Genability and OpenEI -uv run tariff-fetch.cli \ +# Scripted run for Genability +uv run tariff-fetch \ --state ca \ - --providers genability \ - --providers openei \ + --provider genability \ + --effective-date 2025-06-01 \ --output-folder data/exports ``` @@ -123,7 +141,7 @@ can accept or override them. For direct conversion of a single Arcadia master tariff to URDB JSON: ```bash -uv run tariff-fetch-arcadia-urdb 522 2025 +uv run tariff-fetch urdb ni 522 2025 ``` Useful options: @@ -131,4 +149,130 @@ Useful options: - `--output` / `-o`: output file path - `--apply-percentages` / `--no-apply-percentages` - `--charge-class`: repeat to include multiple charge classes +- `--cc`: compact charge-class selector using `S T D t C U A O N n` +- `--property`: repeat `key=value` to pre-fill Arcadia tariff properties - `--force` / `-f`: overwrite an existing output file + +Arcadia property overrides accept either the machine-readable property key or the user-facing property name. For +CHOICE properties, the value can be either the Arcadia option value or the user-facing choice label. + +Example: + +```bash +uv run tariff-fetch urdb ni 522 2025 \ + --property territoryId=123 \ + --property "Territory=Primary Territory" +``` + +## Direct Arcadia Raw Fetch + +Fetch a single Arcadia master tariff as raw JSON without going through the interactive utility picker: + +```bash +uv run tariff-fetch ni arcadia 522 +uv run tariff-fetch ni arcadia 522 2025-06-01 +``` + +If `EFFECTIVE_DATE` is omitted, the command uses today. + +Useful options: + +- `--output` / `-o`: output file path +- `--force` / `-f`: overwrite an existing output file +- `--log-dir`: directory for log files +- `--log-file`: exact log file path + +## Direct RateAcuity Fetch + +RateAcuity does not expose a stable tariff identifier like Arcadia's `master_tariff_id`, so the non-interactive +commands work by fuzzy-matching your input against the live dropdown choices shown in the RateAcuity web portal at +runtime. + +Available commands: + +```bash +uv run tariff-fetch ni rateacuity fuzzy ny "con ed" --tariff "residential service" +uv run tariff-fetch ni rateacuity eia-id 123 --tariff "residential service" +uv run tariff-fetch gas ni ny "con ed gas" --tariff "firm gas service" +uv run tariff-fetch gas urdb ni ny "con ed gas" --year 2025 --tariff "firm gas service" +``` + +### How fuzzy matching works + +- The CLI loads the current RateAcuity utility list for the requested state. +- It lowercases both your query and every available RateAcuity choice before scoring them. +- It picks the highest-scoring utility match. +- After selecting that utility, it loads the current tariff list and fuzzy-matches each `--tariff` query the same way. +- If multiple `--tariff` queries resolve to the same RateAcuity tariff, the duplicate is ignored and that tariff is only fetched once. + +This means your query does not need to be an exact string from the portal. Shortened, lowercased, or partial input is +usually fine. + +Examples: + +```bash +# Utility query does not need to match RateAcuity text exactly +uv run tariff-fetch ni rateacuity fuzzy ny "con ed" --tariff "residential service" + +# Tariff queries are also fuzzy-matched and case-insensitive +uv run tariff-fetch ni rateacuity fuzzy ny "consolidated edison" \ + --tariff "RESIDENTIAL" \ + --tariff "time of use" + +# Electric raw fetch using EIA-based utility lookup from the cached parquet +uv run tariff-fetch ni rateacuity eia-id 123 --tariff "small commercial" +``` + +### Important fuzzy-matching behavior + +- Matching is performed against the live strings that RateAcuity returns in the browser session. +- Matching is case-insensitive because both sides are compared as lowercase. +- The command does not stop to ask "did you mean X?" in non-interactive mode. It chooses the best match and proceeds. +- If your query is too broad, the "best" result may still be the wrong tariff. + +In practice, use queries that are distinctive enough to narrow the target: + +- Better: `"residential service"` +- Riskier: `"residential"` +- Better: `"firm gas service"` +- Riskier: `"service"` + +When you are unsure what RateAcuity calls a tariff, start with the interactive flow once, note the exact names shown in +the dropdowns, and then use those strings in the non-interactive commands. + +### RateAcuity command summary + +- `tariff-fetch ni rateacuity fuzzy`: electric raw fetch by state plus fuzzy utility/tariff matching +- `tariff-fetch ni rateacuity eia-id`: electric raw fetch by utility EIA id, then fuzzy tariff matching +- `tariff-fetch gas ni`: gas raw fetch by state plus fuzzy utility/tariff matching +- `tariff-fetch gas urdb ni`: gas URDB conversion by state plus fuzzy utility/tariff matching + +For `tariff-fetch gas urdb ni`, you must also provide the conversion year and any URDB metadata you want to override: + +```bash +uv run tariff-fetch gas urdb ni ny "con ed gas" \ + --year 2025 \ + --tariff "firm gas service" \ + --label ceg \ + --sector Commercial \ + --servicetype Delivery \ + --apply-percentages +``` + +## Show Arcadia Properties + +Inspect the Arcadia property keys, user-facing names, descriptions, and CHOICE aliases for a master tariff before +running conversion: + +```bash +uv run tariff-fetch show-properties 522 +uv run tariff-fetch show-properties 522 2025-06-01 +``` + +## Cache Management + +Clear the cached utility parquet used by the interactive utility picker: + +```bash +uv run tariff-fetch cache clear +``` diff --git a/docs/cli-usage.md b/docs/cli-usage.md index 39eb374..f2132c0 100644 --- a/docs/cli-usage.md +++ b/docs/cli-usage.md @@ -9,7 +9,13 @@ If you prefer not to clone the repository or manage a local virtual environment, uvx --env-file=.env --from git+https://github.com/switchbox-data/tariff_fetch tariff-fetch # gas -uvx --env-file=.env --from git+https://github.com/switchbox-data/tariff_fetch tariff-fetch-gas +uvx --env-file=.env --from git+https://github.com/switchbox-data/tariff_fetch tariff-fetch gas + +# arcadia to urdb +uvx --env-file=.env --from git+https://github.com/switchbox-data/tariff_fetch tariff-fetch urdb ni 522 2025 + +# raw arcadia tariff by master tariff id +uvx --env-file=.env --from git+https://github.com/switchbox-data/tariff_fetch tariff-fetch ni arcadia 522 2025-06-01 ``` All environment variables (API keys, credentials, etc.) still need to be exported or added to your `.env` file beforehand. @@ -21,28 +27,204 @@ Run `uv run tariff-fetch` (or `python -m tariff_fetch.cli` / `just cli`) to laun ### Options - `--state` / `-s`: two-letter state abbreviation (case-insensitive). If omitted, the CLI prompts you. -- `--providers` / `-p`: repeatable flag for each provider (`genability`, `openei`, `rateacuity`). Leaving it out opens a checkbox prompt so you can select multiple providers at once. +- `--provider` / `-p`: provider to fetch (`genability`, `openei`, `rateacuity`). If omitted, the CLI prompts you. - `--output-folder` / `-o`: directory for exported JSON files. Defaults to `./outputs`. +- `--effective-date`: provider query date in `YYYY-MM-DD` format. +- `--no-input`: fail instead of prompting for interactive input. +- `--log-dir`: directory for log files. +- `--log-file`: exact file path for the log file. ### Workflow Overview 1. Pick a state (option or prompt). -2. Choose which providers to fetch (option or interactive checkbox). +2. Choose which provider to fetch (option or prompt). 3. Select a utility from the structured EIA list. The CLI fetches the latest CORE_EIA861 data to help you pick based on name, entity type, sales, revenue, and customer counts. -4. For each provider selected, `tariff_fetch` runs the corresponding workflow (`process_genability`, `process_openei`, or `process_rateacuity`) and writes exports to the chosen output folder. Authentication failures print guidance about the relevant environment variables. +4. `tariff_fetch` runs the selected workflow (`process_genability`, `process_openei`, or `process_rateacuity`) and writes exports to the chosen output folder. Authentication failures print guidance about the relevant environment variables. + +The utility picker caches the CORE_EIA861 parquet for 1 hour in the platform-specific user cache directory, so repeated runs usually reuse the local copy instead of downloading it again. + +Example: + +```bash +uv run tariff-fetch --state ca --provider genability --effective-date 2025-06-01 +``` + +Use `--no-input` in automation when every required value must come from flags or environment configuration. If the CLI +would otherwise prompt, it exits with code `1` and names the missing prompt. -## Gas CLI (`tariff-fetch-gas`) +## URDB CLI (`tariff-fetch urdb`) -Run `uv run tariff-fetch-gas` (or `python -m tariff_fetch.cli_gas` / `just cligas`). +Run `uv run tariff-fetch urdb` for the interactive Genability-to-URDB flow. + +### Options + +- `--state` / `-s`: two-letter state abbreviation (case-insensitive). If omitted, the CLI prompts you. +- `--output-folder` / `-o`: directory for exported files. Defaults to `./outputs`. +- `--year` / `-y`: year to convert. If omitted, the CLI prompts you. +- `--log-dir`: directory for log files. +- `--log-file`: exact file path for the log file. +- `--fail-fast`: stop immediately on conversion errors instead of prompting to continue. +- `--property`: tariff property override in `key=value` form; repeat to provide multiple values. + +### Subcommands + +- `ni`: convert one Arcadia master tariff directly to URDB JSON. + +Example: + +```bash +uv run tariff-fetch urdb ni 522 2025 --output ./outputs/arcadia_urdb_522_2025.json + +uv run tariff-fetch urdb ni 522 2025 --cc STDtCUAONn +``` + +Arcadia property overrides accept either the canonical property key or the user-facing property name. For CHOICE +properties, values can be either Arcadia option values or user-facing choice labels. + +Example: + +```bash +uv run tariff-fetch urdb ni 522 2025 \ + --property territoryId=123 \ + --property "Territory=Primary Territory" +``` + +## Direct Fetch CLI (`tariff-fetch ni`) + +Use this command to fetch provider data directly by identifier. + +### Subcommands + +- `arcadia`: fetch one Arcadia master tariff as raw JSON. +- `rateacuity fuzzy`: fetch one or more electric RateAcuity tariffs by fuzzy-matched state, utility, and tariff names. +- `rateacuity eia-id`: resolve an electric utility from the cached utility parquet by EIA ID, then fuzzy-match tariff names. + +Examples: + +```bash +uv run tariff-fetch ni arcadia 522 +uv run tariff-fetch ni arcadia 522 2025-06-01 --output ./outputs/arcadia_522_2025-06-01.json +uv run tariff-fetch ni rateacuity fuzzy ny "con ed" --tariff "residential service" +uv run tariff-fetch ni rateacuity eia-id 123 --tariff "small commercial" +``` + +If the effective date is omitted, the command uses today. + +### RateAcuity Fuzzy Matching + +RateAcuity does not offer a stable tariff identifier through this CLI, so the non-interactive commands match against +the live dropdown labels returned by the RateAcuity web portal. + +The matching rules are: + +1. The CLI loads the available utility or tariff strings from RateAcuity at runtime. +2. Your query and each available choice are both lowercased before comparison. +3. The CLI computes a fuzzy score and picks the highest-scoring match. +4. For repeated `--tariff` flags, each query is matched independently and duplicate resolved tariffs are removed. + +That means these inputs are treated similarly: + +```bash +uv run tariff-fetch ni rateacuity fuzzy ny "con ed" --tariff "residential service" +uv run tariff-fetch ni rateacuity fuzzy ny "CON ED" --tariff "RESIDENTIAL SERVICE" +uv run tariff-fetch ni rateacuity fuzzy ny "consolidated edison" --tariff "residential" +``` + +The first two are typically safe because they are distinctive and close to the portal text. The third may still work, +but broader queries are inherently riskier because the command will choose the best-scoring match without pausing for +confirmation. + +Practical guidance: + +- Prefer distinctive phrases over generic fragments. +- Use the interactive RateAcuity flow once if you need to learn the exact utility and tariff naming used in the portal. +- Treat `--tariff "service"` or `--tariff "residential"` as ambiguous unless you know the target list is small. + +Examples: + +```bash +# Electric raw fetch with state + utility query +uv run tariff-fetch ni rateacuity fuzzy ny "con ed" \ + --tariff "residential service" \ + --tariff "time of use" + +# Electric raw fetch with utility resolved from cached parquet by EIA id +uv run tariff-fetch ni rateacuity eia-id 123 \ + --tariff "small commercial" +``` + +## Property Inspection CLI (`tariff-fetch show-properties`) + +Use this command to inspect Arcadia tariff property metadata before conversion. + +Examples: + +```bash +uv run tariff-fetch show-properties 522 +uv run tariff-fetch show-properties 522 2025-06-01 +``` + +The command prints the Arcadia property key, user-facing name, data type, description, and any CHOICE aliases that +can be used with `--property`. + +## Cache CLI (`tariff-fetch cache`) + +Use this command to clear the cached utility parquet used by the interactive utility picker. + +### Subcommands + +- `clear`: remove the cached CORE_EIA861 parquet file. + +Example: + +```bash +uv run tariff-fetch cache clear +``` + +## Gas CLI (`tariff-fetch gas`) + +Run `uv run tariff-fetch gas` (or `python -m tariff_fetch.cli gas` / `just cli`). ### Options - `--state` / `-s`: gas benchmark state (prompts if omitted). - `--output-folder` / `-o`: output directory (defaults to `./outputs`). -- `--urdb`: use this flag to convert to URDB format + +### Subcommands + +- `ni`: fetch gas tariffs non-interactively by fuzzy-matched state, utility, and tariff names. +- `urdb`: convert gas tariffs to URDB format. + +Examples: + +```bash +uv run tariff-fetch gas --state tx --output-folder outputs +uv run tariff-fetch gas ni ny "con ed gas" --tariff "firm gas service" +uv run tariff-fetch gas urdb --state tx --year 2025 --output-folder outputs +uv run tariff-fetch gas urdb ni ny "con ed gas" --year 2025 --tariff "firm gas service" +``` ### Workflow Overview This command only targets RateAcuity’s gas workflow. After you confirm the state, the CLI launches the Selenium flow via `process_rateacuity_gas`, exporting the selected schedules. Failures typically mean the `RATEACUITY_USERNAME`/`RATEACUITY_PASSWORD` credentials or local Chrome/Chromium installation need attention. -You may be presented with additional questions when converting to URDB format. +`tariff-fetch gas urdb` runs the RateAcuity gas-to-URDB flow and may prompt for a year if `--year` is omitted. + +`tariff-fetch gas ni` uses the same lowercase fuzzy-matching behavior as `tariff-fetch ni rateacuity fuzzy`, but targets +the gas benchmark workflow instead of electric rates. + +`tariff-fetch gas urdb ni` applies the same fuzzy utility/tariff matching to the gas history-to-URDB conversion flow. +Because this mode is non-interactive, you must provide the conversion year explicitly, and you can optionally control +URDB metadata with flags like `--label`, `--sector`, `--servicetype`, and `--apply-percentages`. + +Example: + +```bash +uv run tariff-fetch gas urdb ni ny "con ed gas" \ + --year 2025 \ + --tariff "firm gas service" \ + --label ceg \ + --sector Commercial \ + --servicetype Delivery \ + --apply-percentages +``` diff --git a/docs/index.md b/docs/index.md index 57ca58d..03598c1 100644 --- a/docs/index.md +++ b/docs/index.md @@ -20,16 +20,16 @@ cd tariff_fetch uv sync # or: python -m venv .venv && source .venv/bin/activate && pip install -e . # run electricity CLI -uv run tariff-fetch --state ca --providers genability --providers openei --output-folder outputs +uv run tariff-fetch --state ca --provider genability --output-folder outputs # run gas CLI -uv run tariff-fetch-gas --state tx --output-folder outputs +uv run tariff-fetch gas --state tx --output-folder outputs ``` Other entry points: -- `python -m tariff_fetch.cli` / `python -m tariff_fetch.cli_gas` -- `just cli` / `just cligas` (from the repo root) +- `python -m tariff_fetch.cli` +- `just cli` (from the repo root) Populate a `.env` file (or export variables) before running: diff --git a/docs/providers/arcadia/urdb-converter.md b/docs/providers/arcadia/urdb-converter.md index 1a5c5cd..0617b1d 100644 --- a/docs/providers/arcadia/urdb-converter.md +++ b/docs/providers/arcadia/urdb-converter.md @@ -4,6 +4,45 @@ This page explains what the Arcadia-to-URDB converter does, what it asks for, an The goal of the converter is to turn Arcadia electricity tariffs into a URDB-style rate record that is useful for downstream analysis. It is intentionally conservative: if the tariff uses features that are not yet handled safely, the converter should stop instead of silently producing a misleading result. +## CLI entrypoint + +Run the direct converter with: + +```bash +uv run tariff-fetch urdb ni MASTER_TARIFF_ID YEAR [OPTIONS] +``` + +Example: + +```bash +uv run tariff-fetch urdb ni 522 2025 --output ./outputs/arcadia_urdb_522_2025.json +``` + +Before conversion, you can inspect the tariff's available Arcadia properties with: + +```bash +uv run tariff-fetch show-properties 522 +uv run tariff-fetch show-properties 522 2025-06-01 +``` + +You can also pre-fill Arcadia tariff properties from the command line: + +```bash +uv run tariff-fetch urdb ni 522 2025 \ + --property territoryId=123 \ + --property "Territory=Primary Territory" +``` + +Property overrides accept either: + +- the canonical Arcadia property key such as `territoryId` +- the user-facing property name such as `Territory` + +For CHOICE properties, values can be either: + +- the Arcadia machine value +- the user-facing choice label shown in interactive prompts + ## What it converts The converter is designed for Arcadia electricity tariffs with: diff --git a/docs/providers/rateacuity/index.md b/docs/providers/rateacuity/index.md index fdc71d3..7dda98a 100644 --- a/docs/providers/rateacuity/index.md +++ b/docs/providers/rateacuity/index.md @@ -14,6 +14,54 @@ Rate Acuity provides US utility rate data for both electricity and gas through t COMING SOON. +## CLI Modes + +`tariff_fetch` supports both interactive and non-interactive RateAcuity workflows. + +Non-interactive commands: + +```bash +uv run tariff-fetch ni rateacuity fuzzy ny "con ed" --tariff "residential service" +uv run tariff-fetch ni rateacuity eia-id 123 --tariff "small commercial" +uv run tariff-fetch gas ni ny "con ed gas" --tariff "firm gas service" +uv run tariff-fetch gas urdb ni ny "con ed gas" --year 2025 --tariff "firm gas service" +``` + +The electric raw workflow has two modes: + +- `ni rateacuity fuzzy`: you provide the state and a utility query string +- `ni rateacuity eia-id`: the CLI resolves the utility name from the cached parquet using an EIA ID, then proceeds + +Gas currently supports fuzzy matching only: + +- `gas ni`: raw gas fetch +- `gas urdb ni`: gas history to URDB conversion + +## Fuzzy Matching + +RateAcuity choices are only known at runtime because they come from the live web portal dropdowns. For that reason, +the non-interactive commands do not accept a stable tariff id. Instead, they fuzzy-match your input against the live +utility and schedule labels. + +Behavior: + +- both your query and the available RateAcuity choices are lowercased before comparison +- the best fuzzy match is selected automatically +- repeated `--tariff` queries that resolve to the same schedule are deduplicated + +Examples: + +```bash +# likely matches "Consolidated Edison Company of New York" +uv run tariff-fetch ni rateacuity fuzzy ny "con ed" --tariff "residential service" + +# uppercase/lowercase differences do not matter +uv run tariff-fetch ni rateacuity fuzzy ny "CON ED" --tariff "RESIDENTIAL" +``` + +Be careful with broad tariff fragments such as `"service"` or `"residential"`. The non-interactive commands do not ask +for confirmation; they select the best match and continue. + ## Features Rate Acuity offers: diff --git a/pyproject.toml b/pyproject.toml index 475ac39..f026beb 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -11,6 +11,7 @@ dependencies = [ "fuzzywuzzy>=0.18.0", "pathvalidate>=3.3.1", "polars>=1.34.0", + "platformdirs>=4.5.0", "pydantic>=2.12.5", "python-dotenv>=1.1.1", "python-levenshtein>=0.27.1", @@ -38,8 +39,6 @@ Documentation = "https://switchbox-data.github.io/tariff_fetch/" [project.scripts] tariff-fetch = "tariff_fetch.cli:main_cli" -tariff-fetch-gas = "tariff_fetch.cli_gas:main_cli" -tariff-fetch-arcadia-urdb = "tariff_fetch.cli_arcadia_urdb:main_cli" [dependency-groups] dev = [ diff --git a/tariff_fetch/_cli/__init__.py b/tariff_fetch/_cli/__init__.py index aa204ae..3c6a05f 100644 --- a/tariff_fetch/_cli/__init__.py +++ b/tariff_fetch/_cli/__init__.py @@ -1,16 +1,31 @@ import os from datetime import datetime from pathlib import Path -from typing import cast -import questionary from pathvalidate import sanitize_filename from rich.console import Console +from tariff_fetch import questionary_typed as q + console = Console() -def prompt_filename(output_folder: Path, suggested_filename: str, extension: str) -> Path: +def _json_file_filter(path: str) -> bool: + return Path(path).suffix == ".json" + + +def _file_filter_for_extension(extension: str): + def _file_filter(path: str) -> bool: + return Path(path).suffix == f".{extension}" + + return _file_filter + + +def _validate_new_path(path: str) -> bool | str: + return (not os.path.exists(path)) or "A file with that name already exists" + + +def prompt_filename(output_folder: Path, suggested_filename: str, extension: str) -> Path | None: date_str = datetime.now().strftime("%Y-%m-%d") suggested_filename = sanitize_filename(f"{suggested_filename}_{date_str}") if output_folder.exists(): @@ -24,14 +39,13 @@ def prompt_filename(output_folder: Path, suggested_filename: str, extension: str else: filepath = output_folder.joinpath(f"{suggested_filename}-0{os.extsep}{extension}") - return Path( - cast( - str, - questionary.path( - message="Path to save the results", - default=filepath.as_posix(), - file_filter=lambda _: Path(_).suffix == extension, - validate=lambda _: (not os.path.exists(_)) or "A file with that name already exists", # pyright: ignore[reportUnknownLambdaType, reportUnknownArgumentType] - ).ask(), - ) + result = q.path( + message="Path to save the results", + default=filepath.as_posix(), + file_filter=_json_file_filter if extension == "json" else _file_filter_for_extension(extension), + validate=_validate_new_path, ) + value = result.ask() + if value is None: + return None + return Path(value) diff --git a/tariff_fetch/_cli/arcadia_urdb.py b/tariff_fetch/_cli/arcadia_urdb.py index 86652c7..4d35f14 100644 --- a/tariff_fetch/_cli/arcadia_urdb.py +++ b/tariff_fetch/_cli/arcadia_urdb.py @@ -1,36 +1,50 @@ import json -import os +import shlex from datetime import date from pathlib import Path -from typing import cast +from typing import cast, get_args -import questionary from dotenv import load_dotenv +from tariff_fetch import questionary_typed as q + # from tariff_fetch.genability.lse import get_lses_page # from tariff_fetch.genability.tariffs import CustomerClass, TariffType, tariffs_paginate from tariff_fetch.arcadia.api import ArcadiaSignalAPI -from tariff_fetch.arcadia.schema.common import CustomerClass, TariffType +from tariff_fetch.arcadia.schema.common import CustomerClass, RateChargeClass, TariffType from tariff_fetch.arcadia.schema.tariff import TariffExtended +from tariff_fetch.arcadia.schema.tariffproperty import TariffPropertyStandard from tariff_fetch.urdb.arcadia.build import build_urdb from tariff_fetch.urdb.arcadia.prompts import prompt_charge_classes -from tariff_fetch.urdb.arcadia.scenario import Scenario +from tariff_fetch.urdb.arcadia.scenario import Scenario, ScenarioPropertyValue from tariff_fetch.urdb.schema import URDBRate from . import console, prompt_filename from .types import Utility +_ALL_CHARGE_CLASSES = cast(tuple[RateChargeClass, ...], get_args(RateChargeClass)) +_CHARGE_CLASS_CODES: tuple[tuple[str, RateChargeClass], ...] = ( + ("S", "SUPPLY"), + ("T", "TRANSMISSION"), + ("D", "DISTRIBUTION"), + ("t", "TAX"), + ("C", "CONTRACTED"), + ("U", "USER_ADJUSTED"), + ("A", "AFTER_TAX"), + ("O", "OTHER"), + ("N", "NON_BYPASSABLE"), + ("n", "NET_EXCESS"), +) + -def process_genability(utility: Utility, output_folder: Path, year: int): +def process_genability( + utility: Utility, + output_folder: Path, + year: int, + interactive_errors: bool, + properties: dict[str, ScenarioPropertyValue] | None = None, +): _ = load_dotenv() - if not os.getenv("ARCADIA_APP_ID"): - console.print("[b]ARCADIA_APP_ID[/] environment variable is not set.") - if not os.getenv("ARCADIA_APP_KEY"): - console.print("[b]ARCADIA_APP_KEY[/] environment variable is not set.") - if not (os.getenv("ARCADIA_APP_ID") and os.getenv("ARCADIA_APP_KEY")): - console.print("Cannot use Arcadia API due to missing credentials") - _ = console.input("Press enter to proceed...") - return api = ArcadiaSignalAPI() lse_id = _find_utility_lse_id(api, utility) @@ -50,12 +64,11 @@ def process_genability(utility: Utility, output_folder: Path, year: int): api_results = _fetch_tariffs(api, tariffs, year) results: list[URDBRate] = [] + replay_commands: list[str] = [] # tariff_ids = {id_ for _, id_ in tariffs} - apply_percentages = cast(bool | None, questionary.confirm("Apply percentage rates?").ask()) - if apply_percentages is None: - return + apply_percentages = q.confirm("Apply percentage rates?").ask_or_exit() for tariff in api_results: tariff_name = tariff["tariff_name"] @@ -68,11 +81,13 @@ def process_genability(utility: Utility, output_folder: Path, year: int): year=year, apply_percentages=apply_percentages, charge_classes=charge_classes, + properties=properties or {}, ) - urdb_tariff = build_urdb(api, scenario) + urdb_tariff = build_urdb(api, scenario, interactive_errors=interactive_errors) urdb_tariff["name"] = _prompt_tariff_name(urdb_tariff.get("name", "")) results.append(urdb_tariff) + replay_commands.append(_format_replay_command(scenario, tariff)) # results.append(build_urdb(tariff, api, scenario)) suggested_filename = f"arcadia_{utility.name}.urdb.{year}." @@ -83,6 +98,10 @@ def process_genability(utility: Utility, output_folder: Path, year: int): filename.parent.mkdir(exist_ok=True) _ = filename.write_text(json.dumps({"items": results}, indent=2)) console.print(f"Wrote [blue]{len(results)}[/] records to {filename}") + if replay_commands: + console.print("Replay with `tariff-fetch urdb ni`:") + for command in replay_commands: + console.print(command) def _find_utility_lse_id(api: ArcadiaSignalAPI, utility: Utility) -> int | None: @@ -106,15 +125,14 @@ def _find_utility_lse_id(api: ArcadiaSignalAPI, utility: Utility) -> int | None: return utility_lse_id else: # Nothing found; this should *theoretically* never happen but let's keep it just in case - choices = [questionary.Choice(title=_["name"], value=_["lse_id"]) for _ in lses] - choices.append(questionary.Separator()) - choices.append(questionary.Choice(title="None of these", value=None)) - utility_lse_id = cast( - int | None, - questionary.select( - message=f"Found multiple utilities with lse id = {utility.eia_id}. Select one.", choices=choices - ).ask(), - ) + choices: list[q.Choice[int | None] | q.Separator] = [ + q.Choice(title=lse["name"], value=lse["lse_id"]) for lse in lses + ] + choices.append(q.Separator()) + choices.append(q.Choice(title="None of these", value=None)) + utility_lse_id = q.select( + message=f"Found multiple utilities with lse id = {utility.eia_id}. Select one.", choices=choices + ).ask() if utility_lse_id is None: console.print("No utility chosen") return None @@ -135,60 +153,56 @@ def _select_tariffs( ) if not tariffs: return [] - return cast( - list[tuple[str, int]], - questionary.checkbox( - message="Select tariffs", - choices=[ - questionary.Choice( - title=f"{_['tariff_name']} ({_['tariff_id']})", - value=(_["tariff_name"], _["master_tariff_id"]), # pyright: ignore[reportAny] - checked=True, - ) - for _ in tariffs - ], - use_search_filter=True, - use_jk_keys=False, - ).ask(), - ) + result = q.checkbox( + message="Select tariffs", + choices=[ + q.Choice( + title=f"{tariff_['tariff_name']} ({tariff_['master_tariff_id']})", + value=(tariff_["tariff_name"], tariff_["master_tariff_id"]), + checked=True, + ) + for tariff_ in tariffs + ], + use_search_filter=True, + use_jk_keys=False, + ).ask_or_exit() + return result def _select_customer_classes() -> list[CustomerClass]: - return cast( - list[CustomerClass], - questionary.checkbox( - message="Select customer classes", - choices=[ - questionary.Choice(title="Residential", value="RESIDENTIAL"), - questionary.Choice(title="General", value="GENERAL"), - questionary.Choice(title="Special Use", value="SPECIAL_USE"), - ], - validate=lambda _: True if _ else "Select at least one customer class", - ).ask(), - ) + choices: list[q.Choice[CustomerClass]] = [ + q.Choice(title="Residential", value="RESIDENTIAL"), + q.Choice(title="General", value="GENERAL"), + q.Choice(title="Special Use", value="SPECIAL_USE"), + ] + result = q.checkbox( + message="Select customer classes", + choices=choices, + validate=lambda items: True if items else "Select at least one customer class", + ).ask_or_exit() + return result def _select_tariff_types() -> list[TariffType]: - return cast( - list[TariffType], - questionary.checkbox( - message="Select tariff types", - choices=[ - questionary.Choice(title="Default", value="DEFAULT"), - questionary.Choice(title="Alternative", value="ALTERNATIVE"), - questionary.Choice(title="Optional extra", value="OPTIONAL_EXTRA"), - questionary.Choice(title="Rider", value="RIDER"), - ], - validate=lambda _: bool(_) or "Select at least one tariff type", - ).ask(), - ) + choices: list[q.Choice[TariffType]] = [ + q.Choice(title="Default", value="DEFAULT"), + q.Choice(title="Alternative", value="ALTERNATIVE"), + q.Choice(title="Optional extra", value="OPTIONAL_EXTRA"), + q.Choice(title="Rider", value="RIDER"), + ] + result = q.checkbox( + message="Select tariff types", + choices=choices, + validate=lambda items: bool(items) or "Select at least one tariff type", + ).ask_or_exit() + return result def _fetch_tariffs(api: ArcadiaSignalAPI, tariffs: list[tuple[str, int]], year: int): result: list[TariffExtended] = [] with console.status("Fetching tariffs..."): for name, id_ in tariffs: - console.print(f"Tariff id: {name}") + console.print(f"Master tariff id: {id_} ({name})") page = api.tariffs.iter_pages( fields="ext", master_tariff_id=id_, @@ -201,7 +215,75 @@ def _fetch_tariffs(api: ArcadiaSignalAPI, tariffs: list[tuple[str, int]], year: def _prompt_tariff_name(default: str) -> str: - result = cast(str | None, questionary.text("Tariff name", default=default).ask()) - if result is None: - exit() - return result + return q.text("Tariff name", default=default).ask_or_exit() + + +def _format_replay_command(scenario: Scenario, tariff: TariffExtended) -> str: + parts = ["tariff-fetch", "urdb", "ni", str(scenario.master_tariff_id), str(scenario.year)] + if not scenario.apply_percentages: + parts.append("--no-apply-percentages") + + charge_class_flags = _format_charge_class_flags(scenario.charge_classes) + parts.extend(charge_class_flags) + + for key, value in sorted(_canonicalize_properties(scenario.properties, tariff).items()): + values = value if isinstance(value, list) else [value] + for item in values: + parts.extend(["--property", f"{key}={_format_property_value(item)}"]) + + return shlex.join(parts) + + +def _format_charge_class_flags(charge_classes: set[RateChargeClass]) -> list[str]: + if charge_classes == set(_ALL_CHARGE_CLASSES): + return [] + + shortcut = "".join(code for code, charge_class in _CHARGE_CLASS_CODES if charge_class in charge_classes) + return ["-cc", shortcut] + + +def _canonicalize_properties( + properties: dict[str, ScenarioPropertyValue], tariff: TariffExtended +) -> dict[str, ScenarioPropertyValue]: + canonical: dict[str, ScenarioPropertyValue] = {} + consumed_keys: set[str] = set() + property_defs = tariff.get("properties", []) + + for tariff_property in property_defs: + matches = _find_matching_property_keys(properties, tariff_property) + key_name = tariff_property["key_name"] + if key_name in properties: + canonical[key_name] = properties[key_name] + consumed_keys.add(key_name) + continue + if matches: + canonical[key_name] = properties[matches[0]] + consumed_keys.update(matches) + + for key, value in properties.items(): + if key not in consumed_keys and key not in canonical: + canonical[key] = value + return canonical + + +def _find_matching_property_keys( + properties: dict[str, ScenarioPropertyValue], tariff_property: TariffPropertyStandard +) -> list[str]: + key_name = tariff_property["key_name"] + display_name = tariff_property["display_name"] + aliases = {_normalize_property_alias(key_name), _normalize_property_alias(display_name)} + return [candidate for candidate in properties if _normalize_property_alias(candidate) in aliases] + + +def _normalize_property_alias(value: str) -> str: + return "".join(char for char in value.lower() if char.isalnum()) + + +def _format_property_value(value: ScenarioPropertyValue) -> str: + if isinstance(value, bool): + return "true" if value else "false" + if isinstance(value, date): + return value.isoformat() + if isinstance(value, float) and value.is_integer(): + return str(int(value)) + return str(value) diff --git a/tariff_fetch/_cli/genability.py b/tariff_fetch/_cli/genability.py index 360b138..9dec0e5 100644 --- a/tariff_fetch/_cli/genability.py +++ b/tariff_fetch/_cli/genability.py @@ -1,12 +1,13 @@ import os +import shlex from datetime import date from pathlib import Path -from typing import cast -import questionary from dotenv import load_dotenv from pydantic import TypeAdapter +from tariff_fetch import questionary_typed as q + # from tariff_fetch.genability.lse import get_lses_page # from tariff_fetch.genability.tariffs import CustomerClass, TariffType, tariffs_paginate from tariff_fetch.arcadia.api import ArcadiaSignalAPI @@ -38,15 +39,14 @@ def _find_utility_lse_id(api: ArcadiaSignalAPI, utility: Utility) -> int | None: return utility_lse_id else: # Nothing found; this should *theoretically* never happen but let's keep it just in case - choices = [questionary.Choice(title=_["name"], value=_["lse_id"]) for _ in lses] - choices.append(questionary.Separator()) - choices.append(questionary.Choice(title="None of these", value=None)) - utility_lse_id = cast( - int | None, - questionary.select( - message=f"Found multiple utilities with lse id = {utility.eia_id}. Select one.", choices=choices - ).ask(), - ) + choices: list[q.Choice[int | None] | q.Separator] = [ + q.Choice(title=lse["name"], value=lse["lse_id"]) for lse in lses + ] + choices.append(q.Separator()) + choices.append(q.Choice(title="None of these", value=None)) + utility_lse_id = q.select( + message=f"Found multiple utilities with lse id = {utility.eia_id}. Select one.", choices=choices + ).ask() if utility_lse_id is None: console.print("No utility chosen") return None @@ -54,78 +54,77 @@ def _find_utility_lse_id(api: ArcadiaSignalAPI, utility: Utility) -> int | None: def _select_tariffs( - api: ArcadiaSignalAPI, lse_id: int, customer_classes: list[CustomerClass], tariff_types: list[TariffType] + api: ArcadiaSignalAPI, + lse_id: int, + customer_classes: list[CustomerClass], + tariff_types: list[TariffType], + effective_on: date, ) -> list[tuple[str, int]]: with console.status("Fetching tariffs..."): tariffs = list( api.tariffs.iter_pages( lse_id=lse_id, - effective_on=date.today(), + effective_on=effective_on, customer_classes=customer_classes, tariff_types=tariff_types, ) ) if not tariffs: return [] - return cast( - list[tuple[str, int]], - questionary.checkbox( - message="Select tariffs", - choices=[ - questionary.Choice( - title=f"{_['tariff_name']} ({_['tariff_id']})", - value=(_["tariff_name"], _["master_tariff_id"]), # pyright: ignore[reportAny] - checked=True, - ) - for _ in tariffs - ], - use_search_filter=True, - use_jk_keys=False, - ).ask(), - ) + result = q.checkbox( + message="Select tariffs", + choices=[ + q.Choice( + title=f"{tariff_['tariff_name']} ({tariff_['master_tariff_id']})", + value=(tariff_["tariff_name"], tariff_["master_tariff_id"]), + checked=True, + ) + for tariff_ in tariffs + ], + use_search_filter=True, + use_jk_keys=False, + ).ask_or_exit() + return result def _select_customer_classes() -> list[CustomerClass]: - return cast( - list[CustomerClass], - questionary.checkbox( - message="Select customer classes", - choices=[ - questionary.Choice(title="Residential", value="RESIDENTIAL"), - questionary.Choice(title="General", value="GENERAL"), - questionary.Choice(title="Special Use", value="SPECIAL_USE"), - ], - validate=lambda _: True if _ else "Select at least one customer class", - ).ask(), - ) + choices: list[q.Choice[CustomerClass]] = [ + q.Choice(title="Residential", value="RESIDENTIAL"), + q.Choice(title="General", value="GENERAL"), + q.Choice(title="Special Use", value="SPECIAL_USE"), + ] + result = q.checkbox( + message="Select customer classes", + choices=choices, + validate=lambda items: True if items else "Select at least one customer class", + ).ask_or_exit() + return result def _select_tariff_types() -> list[TariffType]: - return cast( - list[TariffType], - questionary.checkbox( - message="Select tariff types", - choices=[ - questionary.Choice(title="Default", value="DEFAULT"), - questionary.Choice(title="Alternative", value="ALTERNATIVE"), - questionary.Choice(title="Optional extra", value="OPTIONAL_EXTRA"), - questionary.Choice(title="Rider", value="RIDER"), - ], - validate=lambda _: bool(_) or "Select at least one tariff type", - ).ask(), - ) - - -def _fetch_tariffs(api: ArcadiaSignalAPI, tariffs: list[tuple[str, int]]): + choices: list[q.Choice[TariffType]] = [ + q.Choice(title="Default", value="DEFAULT"), + q.Choice(title="Alternative", value="ALTERNATIVE"), + q.Choice(title="Optional extra", value="OPTIONAL_EXTRA"), + q.Choice(title="Rider", value="RIDER"), + ] + result = q.checkbox( + message="Select tariff types", + choices=choices, + validate=lambda items: bool(items) or "Select at least one tariff type", + ).ask_or_exit() + return result + + +def _fetch_tariffs(api: ArcadiaSignalAPI, tariffs: list[tuple[str, int]], effective_on: date): result: list[tariff.TariffExtended] = [] with console.status("Fetching tariffs..."): for name, id_ in tariffs: - console.print(f"Tariff id: {name}") + console.print(f"Master tariff id: {id_} ({name})") page = api.tariffs.iter_pages( fields="ext", master_tariff_id=id_, - # effective_on=date.today(), - effective_on=date(2025, 6, 1), + effective_on=effective_on, populate_properties=True, populate_rates=True, ) @@ -133,7 +132,7 @@ def _fetch_tariffs(api: ArcadiaSignalAPI, tariffs: list[tuple[str, int]]): return result -def process_genability(utility: Utility, output_folder: Path): +def process_genability(utility: Utility, output_folder: Path, effective_on: date | None = None): _ = load_dotenv() if not os.getenv("ARCADIA_APP_ID"): console.print("[b]ARCADIA_APP_ID[/] environment variable is not set.") @@ -144,6 +143,7 @@ def process_genability(utility: Utility, output_folder: Path): _ = console.input("Press enter to proceed...") return api = ArcadiaSignalAPI() + effective_on = effective_on or date.today() lse_id = _find_utility_lse_id(api, utility) if lse_id is None: @@ -155,12 +155,20 @@ def process_genability(utility: Utility, output_folder: Path): if not (tariff_types := _select_tariff_types()): return - if not (tariffs := _select_tariffs(api, lse_id, customer_classes, tariff_types)): + if not (tariffs := _select_tariffs(api, lse_id, customer_classes, tariff_types, effective_on)): console.print("[red]No tariffs found[/]") _ = console.input("Press enter to proceed...") return - results = _fetch_tariffs(api, tariffs) + console.print("Replay with `tariff-fetch ni arcadia`:") + for replay_command in _format_replay_commands_from_ids( + [master_tariff_id for _, master_tariff_id in tariffs], effective_on + ): + console.print(replay_command) + if not q.confirm("Proceed?").ask_or_exit(): + return + + results = _fetch_tariffs(api, tariffs, effective_on) suggested_filename = f"arcadia_{utility.name}" if not (filename := prompt_filename(output_folder, suggested_filename, "json")): @@ -169,3 +177,10 @@ def process_genability(utility: Utility, output_folder: Path): filename.parent.mkdir(exist_ok=True) _ = filename.write_bytes(TypeAdapter(list[tariff.TariffExtended]).dump_json(results, indent=2)) console.print(f"Wrote [blue]{len(results)}[/] records to {filename}") + + +def _format_replay_commands_from_ids(master_tariff_ids: list[int], effective_on: date) -> list[str]: + return [ + shlex.join(["tariff-fetch", "ni", "arcadia", str(master_tariff_id), effective_on.isoformat()]) + for master_tariff_id in master_tariff_ids + ] diff --git a/tariff_fetch/_cli/openei.py b/tariff_fetch/_cli/openei.py index 99f326c..f7ebad8 100644 --- a/tariff_fetch/_cli/openei.py +++ b/tariff_fetch/_cli/openei.py @@ -1,12 +1,13 @@ import json import os -from datetime import UTC, datetime +import shlex +from datetime import UTC, date, datetime from pathlib import Path from typing import Literal, cast -import questionary from dotenv import load_dotenv +from tariff_fetch import questionary_typed as q from tariff_fetch.openei.utility_rates import UtilityRateSector, UtilityRatesResponseItem, iter_utility_rates from . import console, prompt_filename @@ -14,32 +15,28 @@ def _prompt_sector() -> UtilityRateSector: - return cast( - UtilityRateSector, - questionary.select( - message="Select sector", - choices=[ - "Residential", - "Commercial", - "Industrial", - "Lighting", - ], - ).ask(), - ) + result = q.select( + message="Select sector", + choices=[ + "Residential", + "Commercial", + "Industrial", + "Lighting", + ], + ).ask_or_exit() + return cast(UtilityRateSector, result) def _prompt_detail_level() -> Literal["full", "minimal"]: - return cast( - Literal["full", "minimal"], - questionary.select( - message="Select level of detail", - choices=["full", "minimal"], - ).ask(), - ) + result = q.select( + message="Select level of detail", + choices=["full", "minimal"], + ).ask_or_exit() + return cast(Literal["full", "minimal"], result) def _get_tariffs( - eia_id: int, sector: UtilityRateSector, detail: Literal["full", "minimal"] + eia_id: int, sector: UtilityRateSector, detail: Literal["full", "minimal"], effective_on: date | None = None ) -> list[UtilityRatesResponseItem]: api_key = os.getenv("OPENEI_API_KEY") if not api_key: @@ -47,7 +44,9 @@ def _get_tariffs( with console.status("Fetching rates..."): iterator = iter_utility_rates( api_key, - effective_on_date=datetime.now(UTC), + effective_on_date=datetime.combine( + effective_on or datetime.now(UTC).date(), datetime.min.time(), tzinfo=UTC + ), sector=sector, detail=detail, eia=eia_id, @@ -56,16 +55,14 @@ def _get_tariffs( def _prompt_tariffs(tariffs: list[UtilityRatesResponseItem]) -> list[UtilityRatesResponseItem]: - return cast( - list[UtilityRatesResponseItem], - questionary.checkbox( - message="Select tariffs to include", - choices=[questionary.Choice(title=_["name"], value=_, checked=True) for _ in tariffs], - ).ask(), - ) + result = q.checkbox( + message="Select tariffs to include", + choices=[q.Choice(title=tariff["name"], value=tariff, checked=True) for tariff in tariffs], + ).ask() + return result or [] -def process_openei(utility: Utility, output_folder: Path): +def process_openei(utility: Utility, output_folder: Path, effective_on: date | None = None): _ = load_dotenv() if not os.getenv("OPENEI_API_KEY"): console.print("[b]OPENEI_API_KEY[/] environment variable is not set") @@ -77,7 +74,7 @@ def process_openei(utility: Utility, output_folder: Path): return if not (detail_level := _prompt_detail_level()): return - tariffs = _get_tariffs(utility.eia_id, sector, detail_level) + tariffs = _get_tariffs(utility.eia_id, sector, detail_level, effective_on) if not tariffs: console.print("[red]No tariffs found[/]") _ = console.input("Press enter to proceed...") @@ -93,6 +90,36 @@ def process_openei(utility: Utility, output_folder: Path): return filepath.parent.mkdir(exist_ok=True) - print(filepath) - _ = filepath.write_text(json.dumps(tariffs, indent=2)) + wrapped_items = {"items": tariffs} + _ = filepath.write_text(json.dumps(wrapped_items, indent=2)) console.print(f"Wrote [blue]{len(tariffs)}[/] items to {filepath}") + console.print("Replay with `tariff-fetch ni openei`:") + for replay_command in _format_replay_commands(utility.eia_id, sector, detail_level, effective_on, tariffs): + console.print(replay_command) + + +def _format_replay_commands( + eia_id: int, + sector: UtilityRateSector, + detail: Literal["full", "minimal"], + effective_on: date | None, + tariffs: list[UtilityRatesResponseItem], +) -> list[str]: + effective_date = (effective_on or datetime.now(UTC).date()).isoformat() + return [ + shlex.join( + [ + "tariff-fetch", + "ni", + "openei", + str(eia_id), + sector, + effective_date, + "--detail", + detail, + "--label", + tariff["label"], + ] + ) + for tariff in tariffs + ] diff --git a/tariff_fetch/_cli/rateacuity.py b/tariff_fetch/_cli/rateacuity.py index 51bf365..50caa12 100644 --- a/tariff_fetch/_cli/rateacuity.py +++ b/tariff_fetch/_cli/rateacuity.py @@ -1,20 +1,50 @@ import json +import logging import os +import shlex +from collections.abc import Sequence from pathlib import Path -from typing import cast -import questionary import tenacity from dotenv import load_dotenv from fuzzywuzzy import fuzz # pyright: ignore[reportMissingTypeStubs] from selenium.common.exceptions import WebDriverException +from tariff_fetch import questionary_typed as q from tariff_fetch._cli.types import Utility from tariff_fetch.rateacuity import LoginState, create_context from tariff_fetch.rateacuity.schema import Tariff from . import console, prompt_filename +logger = logging.getLogger(__name__) + + +def rateacuity_match_score(query: str, choice: str) -> int: + return int(fuzz.ratio(query.lower(), choice.lower())) # pyright: ignore[reportUnknownMemberType] + + +def rank_rateacuity_choices(query: str, choices: Sequence[str]) -> list[str]: + return sorted(choices, key=lambda choice: (rateacuity_match_score(query, choice), choice.lower()), reverse=True) + + +def match_rateacuity_choice(*, query: str, choices: Sequence[str], category: str) -> str: + ranked_choices = rank_rateacuity_choices(query, choices) + if not ranked_choices: + raise RuntimeError(f"RateAcuity shows no {category.lower()} choices for this selection") + match = ranked_choices[0] + logger.info("Matched RateAcuity %s query %r to %r", category.lower(), query, match) + return match + + +def match_rateacuity_choices(*, queries: Sequence[str], choices: Sequence[str], category: str) -> list[str]: + selected_choices: list[str] = [] + for query in queries: + match = match_rateacuity_choice(query=query, choices=choices, category=category) + if match not in selected_choices: + selected_choices.append(match) + return selected_choices + def process_rateacuity_gas(output_folder: Path, state: str): _ = load_dotenv() @@ -45,40 +75,37 @@ def process_rateacuity_gas(output_folder: Path, state: str): raise RuntimeError(f"Something's wrong: rateacuity shows no utilities for this state ({state})") if selected_utility is None: - selected_utility = cast( - str, - questionary.select( - message="Select a utility from available choices", - choices=utilities, - use_jk_keys=False, - use_search_filter=True, - use_shortcuts=False, - ).ask(), - ) - if not selected_utility: - return + selected_utility = q.select( + message="Select a utility from available choices", + choices=utilities, + use_jk_keys=False, + use_search_filter=True, + use_shortcuts=False, + ).ask_or_exit() with console.status("Fetching list of tariffs..."): scraping_state = scraping_state.select_utility(selected_utility) tariffs = [_ for _ in scraping_state.get_schedules() if _] if tariffs_to_include is None: - tariffs_to_include = cast( - list[str], - questionary.checkbox( - message="Select tariffs to include", - choices=tariffs, - use_jk_keys=False, - use_search_filter=True, - validate=lambda _: bool(_) or "Select at least one tariff", - ).ask(), - ) + tariffs_to_include = q.checkbox( + message="Select tariffs to include", + choices=tariffs, + use_jk_keys=False, + use_search_filter=True, + validate=lambda items: bool(items) or "Select at least one tariff", + ).ask_or_exit() if not tariffs_to_include: console.print("[red]No tariffs selected[/]") _ = console.input("Press enter to proceed...") return + console.print("Replay with `tariff-fetch gas ni`:") + console.print(_format_gas_replay_command(state, selected_utility, tariffs_to_include)) + if not q.confirm("Proceed?").ask_or_exit(): + return + with console.status("Fetching tariffs..."): while tariffs_to_include: tariff = tariffs_to_include.pop(0) @@ -90,7 +117,8 @@ def process_rateacuity_gas(output_folder: Path, state: str): assert selected_utility suggested_filename = f"gas_rateacuity_{selected_utility}" - filename = prompt_filename(output_folder, suggested_filename, "json") + if not (filename := prompt_filename(output_folder, suggested_filename, "json")): + return filename.parent.mkdir(exist_ok=True) _ = filename.write_text(json.dumps(results, indent=2)) @@ -124,43 +152,41 @@ def process_rateacuity(output_folder: Path, state: str, utility: Utility): raise RuntimeError(f"Something's wrong: rateacuity shows no utilities for this state ({state})") if selected_utility is None: - utilities_scored = sorted(utilities, key=lambda _: fuzz.ratio(utility.name, _), reverse=True) # pyright: ignore[reportUnknownMemberType] + utilities_scored = rank_rateacuity_choices(utility.name, utilities) selected_utility = utilities_scored.pop(0) - if not questionary.confirm(f"Is this the correct utility: {selected_utility} ?").ask(): - selected_utility = cast( - str, - questionary.select( - message="Select a utility from available choices", - choices=utilities_scored, - use_jk_keys=False, - use_search_filter=True, - use_shortcuts=False, - ).ask(), - ) - if not selected_utility: - return + confirmed = q.confirm(f"Is this the correct utility: {selected_utility} ?").ask_or_exit() + if not confirmed: + selected_utility = q.select( + message="Select a utility from available choices", + choices=utilities_scored, + use_jk_keys=False, + use_search_filter=True, + use_shortcuts=False, + ).ask_or_exit() with console.status("Fetching list of tariffs..."): scraping_state = scraping_state.select_utility(selected_utility) tariffs = [_ for _ in scraping_state.get_schedules() if _] if tariffs_to_include is None: - tariffs_to_include = cast( - list[str], - questionary.checkbox( - message="Select tariffs to include", - choices=tariffs, - use_jk_keys=False, - use_search_filter=True, - validate=lambda _: bool(_) or "Select at least one tariff", - ).ask(), - ) + tariffs_to_include = q.checkbox( + message="Select tariffs to include", + choices=tariffs, + use_jk_keys=False, + use_search_filter=True, + validate=lambda items: bool(items) or "Select at least one tariff", + ).ask_or_exit() if not tariffs_to_include: console.print("[red]No tariffs selected[/]") _ = console.input("Press enter to proceed...") return + console.print("Replay with `tariff-fetch ni rateacuity`:") + console.print(_format_replay_command(utility.eia_id, tariffs_to_include)) + if not q.confirm("Proceed?").ask_or_exit(): + return + with console.status("Fetching tariffs..."): while tariffs_to_include: tariff = tariffs_to_include.pop(0) @@ -176,3 +202,118 @@ def process_rateacuity(output_folder: Path, state: str, utility: Utility): return filename.parent.mkdir(exist_ok=True) _ = filename.write_text(json.dumps(results, indent=2)) + + +def fetch_rateacuity_tariffs( + *, state: str, utility_query: str, tariff_queries: Sequence[str] +) -> tuple[str, list[Tariff]]: + _ = load_dotenv() + username = os.getenv("RATEACUITY_USERNAME") + password = os.getenv("RATEACUITY_PASSWORD") + if not username: + raise ValueError("RATEACUITY_USERNAME environment variable is not set") + if not password: + raise ValueError("RATEACUITY_PASSWORD environment variable is not set") + + selected_utility = "" + results: list[Tariff] = [] + + for attempt in tenacity.Retrying( + stop=tenacity.stop_after_attempt(3), retry=tenacity.retry_if_exception_type(WebDriverException) + ): + with attempt, create_context() as context: + with console.status("Fetching list of utilities..."): + scraping_state = ( + LoginState(context).login(username, password).electric().benchmark_all().select_state(state.upper()) + ) + utilities = [_ for _ in scraping_state.get_utilities() if _] + + if not utilities: + raise RuntimeError(f"Something's wrong: rateacuity shows no utilities for this state ({state})") + + selected_utility = match_rateacuity_choice(query=utility_query, choices=utilities, category="Utility") + + with console.status("Fetching list of tariffs..."): + scraping_state = scraping_state.select_utility(selected_utility) + tariffs = [_ for _ in scraping_state.get_schedules() if _] + + selected_tariffs = match_rateacuity_choices( + queries=tariff_queries, + choices=tariffs, + category="Tariff", + ) + + with console.status("Fetching tariffs..."): + for tariff in selected_tariffs: + console.log(f"Fetching {tariff}") + scraping_state = scraping_state.select_schedule(tariff) + sections = scraping_state.as_sections() + results.append({"schedule": tariff, "sections": sections}) + scraping_state = scraping_state.back_to_selections() + + return selected_utility, results + + +def _format_replay_command(eia_id: int, tariffs: Sequence[str]) -> str: + parts = ["tariff-fetch", "ni", "rateacuity", "eia-id", str(eia_id)] + for tariff in tariffs: + parts.extend(["--tariff", tariff]) + return shlex.join(parts) + + +def _format_gas_replay_command(state: str, utility: str, tariffs: Sequence[str]) -> str: + parts = ["tariff-fetch", "gas", "ni", state] + parts.append(utility) + for tariff in tariffs: + parts.extend(["--tariff", tariff]) + return shlex.join(parts) + + +def fetch_rateacuity_gas_tariffs( + *, state: str, utility_query: str, tariff_queries: Sequence[str] +) -> tuple[str, list[Tariff]]: + _ = load_dotenv() + username = os.getenv("RATEACUITY_USERNAME") + password = os.getenv("RATEACUITY_PASSWORD") + if not username: + raise ValueError("RATEACUITY_USERNAME environment variable is not set") + if not password: + raise ValueError("RATEACUITY_PASSWORD environment variable is not set") + + selected_utility = "" + results: list[Tariff] = [] + + for attempt in tenacity.Retrying( + stop=tenacity.stop_after_attempt(3), retry=tenacity.retry_if_exception_type(WebDriverException) + ): + with attempt, create_context() as context: + with console.status("Fetching list of utilities..."): + scraping_state = ( + LoginState(context).login(username, password).gas().benchmark_all().select_state(state.upper()) + ) + utilities = [_ for _ in scraping_state.get_utilities() if _] + + if not utilities: + raise RuntimeError(f"Something's wrong: rateacuity shows no utilities for this state ({state})") + + selected_utility = match_rateacuity_choice(query=utility_query, choices=utilities, category="Utility") + + with console.status("Fetching list of tariffs..."): + scraping_state = scraping_state.select_utility(selected_utility) + tariffs = [_ for _ in scraping_state.get_schedules() if _] + + selected_tariffs = match_rateacuity_choices( + queries=tariff_queries, + choices=tariffs, + category="Tariff", + ) + + with console.status("Fetching tariffs..."): + for tariff in selected_tariffs: + console.log(f"Fetching {tariff}") + scraping_state = scraping_state.select_schedule(tariff) + sections = scraping_state.as_sections() + results.append({"schedule": tariff, "sections": sections}) + scraping_state = scraping_state.back_to_selections() + + return selected_utility, results diff --git a/tariff_fetch/_cli/rateacuity_gas_urdb.py b/tariff_fetch/_cli/rateacuity_gas_urdb.py index fcc1df7..fc143a1 100644 --- a/tariff_fetch/_cli/rateacuity_gas_urdb.py +++ b/tariff_fetch/_cli/rateacuity_gas_urdb.py @@ -1,17 +1,18 @@ import json import os +import shlex from collections.abc import Collection from datetime import date from pathlib import Path from statistics import mean from typing import cast, get_args -import questionary import tenacity from dotenv import load_dotenv from rich.prompt import Confirm from selenium.common.exceptions import WebDriverException +from tariff_fetch import questionary_typed as q from tariff_fetch.rateacuity import LoginState, create_context from tariff_fetch.urdb.rateacuity_history_gas import ( build_urdb, @@ -20,6 +21,7 @@ from tariff_fetch.urdb.schema import RateSector, ServiceType, URDBRate from . import console, prompt_filename +from .rateacuity import match_rateacuity_choice, match_rateacuity_choices # TODO: This is ungodly ugly but it works @@ -38,6 +40,7 @@ def process_rateacuity_gas_urdb(output_folder: Path, state: str, year: int): selected_utility = None tariffs_to_include = None result: list[URDBRate] = [] + replay_commands: list[str] = [] for attempt in tenacity.Retrying( stop=tenacity.stop_after_attempt(3), retry=tenacity.retry_if_exception_type(WebDriverException) ): @@ -48,32 +51,24 @@ def process_rateacuity_gas_urdb(output_folder: Path, state: str, year: int): ) utilities = [_ for _ in scraping_state.get_utilities() if _] if selected_utility is None: - selected_utility = cast( - str, - questionary.select( - message="Select a utility from available choices", - choices=utilities, - use_jk_keys=False, - use_search_filter=True, - use_shortcuts=False, - ).ask(), - ) - if not selected_utility: - return + selected_utility = q.select( + message="Select a utility from available choices", + choices=utilities, + use_jk_keys=False, + use_search_filter=True, + use_shortcuts=False, + ).ask_or_exit() with console.status("Fetching list of tariffs..."): scraping_state = scraping_state.select_utility(selected_utility) tariffs = [_ for _ in scraping_state.get_schedules() if _] if tariffs_to_include is None: - tariffs_to_include = cast( - list[str], - questionary.checkbox( - message="Select tariffs to include", - choices=tariffs, - use_jk_keys=False, - use_search_filter=True, - validate=lambda _: bool(_) or "Select at least one tariff", - ).ask(), - ) + tariffs_to_include = q.checkbox( + message="Select tariffs to include", + choices=tariffs, + use_jk_keys=False, + use_search_filter=True, + validate=lambda items: bool(items) or "Select at least one tariff", + ).ask_or_exit() if not tariffs_to_include: console.print("[red]No tariffs selected[/]") @@ -118,32 +113,18 @@ def process_rateacuity_gas_urdb(output_folder: Path, state: str, year: int): console.print("Percentages will be applied to the final result as is") apply_percentages = Confirm.ask("Apply percentages? (otherwise percentages will be ignored)") - label = cast( - str | None, questionary.text("Label", default=_utility_name_to_label(selected_utility)).ask() - ) - if label is None: - exit() - sector = cast( - RateSector | None, - questionary.select( - "Sector", - default="Residential", - choices=get_args(RateSector), - ).ask(), - ) - if sector is None: - exit() - - servicetype = cast( - ServiceType | None, - questionary.select( - "Sector", - default="Bundled", - choices=get_args(ServiceType), - ).ask(), - ) - if servicetype is None: - exit() + label = q.text("Label", default=_utility_name_to_label(selected_utility)).ask_or_exit() + sector = q.select( + "Sector", + default="Residential", + choices=get_args(RateSector), + ).ask_or_exit() + + servicetype = q.select( + "Sector", + default="Bundled", + choices=get_args(ServiceType), + ).ask_or_exit() try: urdb = build_urdb(rows, apply_percentages) @@ -153,13 +134,25 @@ def process_rateacuity_gas_urdb(output_folder: Path, state: str, year: int): urdb["utility"] = selected_utility urdb["name"] = tariff urdb["label"] = label - urdb["sector"] = sector - urdb["servicetype"] = servicetype + urdb["sector"] = cast(RateSector, sector) + urdb["servicetype"] = cast(ServiceType, servicetype) urdb["demandunits"] = "kW" urdb["mincharge"] = 0.0 urdb["minchargeunits"] = "$/month" urdb["country"] = "USA" result.append(urdb) + replay_commands.append( + _format_gas_urdb_replay_command( + state=state, + utility=selected_utility, + year=year, + tariff=tariff, + apply_percentages=apply_percentages, + label=label, + sector=cast(RateSector, sector), + servicetype=cast(ServiceType, servicetype), + ) + ) scraping_state = ( scraping_state.back_to_selections() @@ -173,6 +166,99 @@ def process_rateacuity_gas_urdb(output_folder: Path, state: str, year: int): filename.parent.mkdir(exist_ok=True) wrapped_result = {"items": result} _ = filename.write_text(json.dumps(wrapped_result, indent=2)) + if replay_commands: + console.print("Replay with `tariff-fetch gas urdb ni`:") + for command in replay_commands: + console.print(command) + + +def fetch_rateacuity_gas_urdb_rates( + *, + state: str, + utility_query: str, + tariff_queries: Collection[str], + year: int, + apply_percentages: bool, + label: str | None, + sector: RateSector, + servicetype: ServiceType, +) -> tuple[str, list[URDBRate]]: + _ = load_dotenv() + username = os.getenv("RATEACUITY_USERNAME") + password = os.getenv("RATEACUITY_PASSWORD") + if not username: + raise ValueError("RATEACUITY_USERNAME environment variable is not set") + if not password: + raise ValueError("RATEACUITY_PASSWORD environment variable is not set") + + selected_utility = "" + result: list[URDBRate] = [] + for attempt in tenacity.Retrying( + stop=tenacity.stop_after_attempt(3), retry=tenacity.retry_if_exception_type(WebDriverException) + ): + with attempt, create_context() as context: + with console.status("Fetching list of utilities..."): + scraping_state = ( + LoginState(context).login(username, password).gas().history().select_state(state.upper()) + ) + utilities = [_ for _ in scraping_state.get_utilities() if _] + selected_utility = match_rateacuity_choice(query=utility_query, choices=utilities, category="Utility") + with console.status("Fetching list of tariffs..."): + scraping_state = scraping_state.select_utility(selected_utility) + tariffs = [_ for _ in scraping_state.get_schedules() if _] + selected_tariffs = match_rateacuity_choices( + queries=list(tariff_queries), + choices=tariffs, + category="Tariff", + ) + + console.print("Fetching tariffs") + while selected_tariffs: + tariff = selected_tariffs.pop(0) + console.log(f"Fetching {tariff}") + scraping_state = ( + scraping_state.select_schedule(tariff) + .set_enddate(date(year, 12, 1)) + .set_number_of_comparisons(12) + .set_frequency(1) + ) + df = scraping_state.as_dataframe() + hd = HistoryData(df) + validation_errors = hd.validate_rows() + if validation_errors: + console.print("Following rows cannot be processed and will be ignored:") + for error in validation_errors: + console.print(f" - {error.row}") + + if unknown_non_empty_columns := hd.get_unknown_nonempty_columns(): + console.print("Found following unknown non-empty columns. Their values will be ignored:") + for col in unknown_non_empty_columns: + console.print(f" - {col}") + + rows = list(hd.rows()) + try: + urdb = build_urdb(rows, apply_percentages) + except ValueError as e: + raise ValueError(f"Cannot convert tariff {tariff!r} to URDB: {e}") from e + + urdb["utility"] = selected_utility + urdb["name"] = tariff + urdb["label"] = label or _utility_name_to_label(selected_utility) + urdb["sector"] = sector + urdb["servicetype"] = servicetype + urdb["demandunits"] = "kW" + urdb["mincharge"] = 0.0 + urdb["minchargeunits"] = "$/month" + urdb["country"] = "USA" + result.append(urdb) + + scraping_state = ( + scraping_state.back_to_selections() + .history() + .select_state(state.upper()) + .select_utility(selected_utility) + ) + return selected_utility, result def _utility_name_to_label(utility_name: str) -> str: @@ -181,6 +267,30 @@ def _utility_name_to_label(utility_name: str) -> str: return "".join(w[0].lower() for w in utility_name.split() if w) +def _format_gas_urdb_replay_command( + *, + state: str, + utility: str, + year: int, + tariff: str, + apply_percentages: bool, + label: str, + sector: RateSector, + servicetype: ServiceType, +) -> str: + parts = ["tariff-fetch", "gas", "urdb", "ni", state, utility, "--year", str(year), "--tariff", tariff] + if apply_percentages: + parts.append("--apply-percentages") + default_label = _utility_name_to_label(utility) + if label != default_label: + parts.extend(["--label", label]) + if sector != "Residential": + parts.extend(["--sector", sector]) + if servicetype != "Bundled": + parts.extend(["--servicetype", servicetype]) + return shlex.join(parts) + + def _get_percentage_columns(rows: Collection[Row]) -> list[tuple[str, str | None, float]]: return [ (row.rate, row.location, mean(row.month_value_float(month) for month in range(0, 12))) diff --git a/tariff_fetch/arcadia/api.py b/tariff_fetch/arcadia/api.py index 1a3450a..2ceba5c 100644 --- a/tariff_fetch/arcadia/api.py +++ b/tariff_fetch/arcadia/api.py @@ -1,9 +1,10 @@ +import logging import os from collections.abc import Iterator from dataclasses import dataclass, field from datetime import date from json import JSONDecodeError -from typing import Annotated, Any, Generic, Literal, Self, TypeVar, Unpack, overload +from typing import Annotated, Any, Generic, Literal, Self, TypeVar, Unpack, cast, overload from urllib.parse import urljoin import requests @@ -19,6 +20,7 @@ _BASE_URL = "https://api.genability.com/rest/public/" _comma_separated = PlainSerializer(",".join) +logger = logging.getLogger(__name__) _T = TypeVar("_T") _C = TypeVar("_C") @@ -144,11 +146,15 @@ class ArcadiaSignalAPI: session: requests.Session = field(default_factory=requests.Session) def _request(self, path: str, **params) -> dict[str, Any]: # pyright: ignore[reportMissingParameterType, reportUnknownParameterType, reportExplicitAny] + url = urljoin(self.base_url, path) + logger.debug("Arcadia request: GET %s params=%s", url, repr(cast(object, params))) response = self.session.get( - urljoin(self.base_url, path), + url, params=params, # pyright: ignore[reportUnknownArgumentType] auth=HTTPBasicAuth(self.auth.app_id, self.auth.app_key), ) + if response.status_code != 200: + logger.debug("Arcadia response: GET %s status=%s", url, response.status_code) try: response.raise_for_status() except requests.HTTPError as e: diff --git a/tariff_fetch/cli.py b/tariff_fetch/cli.py index c970d0f..bcc72f3 100644 --- a/tariff_fetch/cli.py +++ b/tariff_fetch/cli.py @@ -1,69 +1,883 @@ +import json import logging -from datetime import date +import os +import shutil +from collections.abc import Callable +from datetime import UTC, date, datetime +from enum import Enum from pathlib import Path -from typing import Annotated, cast +from tempfile import NamedTemporaryFile +from typing import Annotated, BinaryIO, Literal, NamedTuple, TypeVar, cast, get_args +from urllib.request import urlopen import polars as pl -import questionary import typer -from requests import HTTPError -from rich.prompt import Prompt +from dotenv import load_dotenv +from pathvalidate import sanitize_filename +from platformdirs import user_cache_dir +from pydantic import TypeAdapter +from rich.logging import RichHandler +from rich.table import Table from tariff_fetch._cli.arcadia_urdb import process_genability as process_genability_urdb from tariff_fetch._cli.genability import process_genability from tariff_fetch._cli.openei import process_openei -from tariff_fetch._cli.rateacuity import process_rateacuity +from tariff_fetch._cli.rateacuity import ( + fetch_rateacuity_gas_tariffs, + fetch_rateacuity_tariffs, + process_rateacuity, + process_rateacuity_gas, +) +from tariff_fetch._cli.rateacuity_gas_urdb import fetch_rateacuity_gas_urdb_rates, process_rateacuity_gas_urdb +from tariff_fetch.arcadia.api import ArcadiaSignalAPI +from tariff_fetch.arcadia.schema import tariff +from tariff_fetch.arcadia.schema.common import RateChargeClass +from tariff_fetch.openei.utility_rates import UtilityRateSector, UtilityRatesResponseItem, iter_utility_rates from tariff_fetch.rateacuity.base import AuthorizationError +from tariff_fetch.urdb.arcadia.build import build_urdb +from tariff_fetch.urdb.arcadia.scenario import Scenario, ScenarioPropertyValue +from . import questionary_typed as q from ._cli import console from ._cli.types import Provider, StateCode, Utility +app = typer.Typer( + add_completion=False, + invoke_without_command=True, + no_args_is_help=False, +) + +urdb_app = typer.Typer( + invoke_without_command=True, + no_args_is_help=False, +) +app.add_typer(urdb_app, name="urdb", help="Convert Arcadia tariffs to URDB JSON.") + +gas_app = typer.Typer( + invoke_without_command=True, + no_args_is_help=False, +) +app.add_typer(gas_app, name="gas", help="Fetch and convert RateAcuity gas tariffs.") + +gas_urdb_app = typer.Typer( + invoke_without_command=True, + no_args_is_help=False, +) +gas_app.add_typer(gas_urdb_app, name="urdb", help="Convert RateAcuity gas tariffs to URDB format.") + +ni_app = typer.Typer( + invoke_without_command=False, + no_args_is_help=True, +) +app.add_typer(ni_app, name="ni", help="Fetch provider data directly by identifier.") + +rateacuity_ni_app = typer.Typer( + invoke_without_command=False, + no_args_is_help=True, +) +ni_app.add_typer( + rateacuity_ni_app, + name="rateacuity", + help="Fetch RateAcuity tariffs in non-interactive modes.", +) + +cache_app = typer.Typer( + invoke_without_command=False, + no_args_is_help=True, +) +app.add_typer(cache_app, name="cache", help="Manage local CLI caches.") + ENTITY_TYPES_SORTORDER = ["Investor Owned", "Cooperative", "Municipal"] CORE_EIA861_YEARLY_SALES_HTTPS = ( "https://s3.us-west-2.amazonaws.com/pudl.catalyst.coop/nightly/core_eia861__yearly_sales.parquet" ) +UTILITY_CACHE_TTL_SECONDS = 60 * 60 +UTILITY_CACHE_DIR = Path(user_cache_dir("tariff_fetch")) +UTILITY_CACHE_PATH = UTILITY_CACHE_DIR / "core_eia861__yearly_sales.parquet" +LOG_FORMAT = "%(asctime)s %(name)s %(levelname)s %(message)s" +ALL_CHARGE_CLASSES = cast(tuple[RateChargeClass, ...], get_args(RateChargeClass)) +CHARGE_CLASS_SHORTCUTS: dict[str, RateChargeClass] = { + "S": "SUPPLY", + "T": "TRANSMISSION", + "D": "DISTRIBUTION", + "t": "TAX", + "C": "CONTRACTED", + "U": "USER_ADJUSTED", + "A": "AFTER_TAX", + "O": "OTHER", + "N": "NON_BYPASSABLE", + "n": "NET_EXCESS", +} +_T = TypeVar("_T") -def prompt_year() -> int: - result = cast( - str, questionary.text("Enter year", default=str(date.today().year - 1), validate=_is_valid_year).ask() +class LogLevel(str, Enum): + DEBUG = "DEBUG" + INFO = "INFO" + WARNING = "WARNING" + ERROR = "ERROR" + CRITICAL = "CRITICAL" + + +@app.callback() +def main_default( + ctx: typer.Context, + state: Annotated[ + StateCode | None, typer.Option("--state", "-s", help="Two-letter state abbreviation", case_sensitive=False) + ] = None, + provider: Annotated[Provider | None, typer.Option("--provider", "-p", case_sensitive=False)] = None, + output_folder: Annotated[ + str, typer.Option("--output-folder", "-o", help="Folder to store outputs in") + ] = "./outputs", + effective_date: Annotated[ + str | None, typer.Option("--effective-date", help="Effective date for provider queries in YYYY-MM-DD format") + ] = None, + log_level: Annotated[ + LogLevel, typer.Option("--log-level", help="Logging level", case_sensitive=False) + ] = LogLevel.INFO, + no_input: Annotated[ + bool, typer.Option("--no-input", help="Fail instead of prompting for interactive input") + ] = False, + log_dir: Annotated[Path | None, typer.Option("--log-dir", help="Directory to write logs to")] = None, + log_file: Annotated[Path | None, typer.Option("--log-file", help="File path to write logs to")] = None, +): + if ctx.invoked_subcommand is not None: + return + _configure_interaction(no_input) + _run_raw( + state, + provider, + output_folder, + _parse_effective_date(effective_date), + _log_level_to_int(log_level), + log_dir, + log_file, ) - return int(result) -def _is_valid_year(value: str) -> bool: +@app.command("raw", help="Fetch raw tariff data from the selected provider.") +def main_raw( + state: Annotated[ + StateCode | None, typer.Option("--state", "-s", help="Two-letter state abbreviation", case_sensitive=False) + ] = None, + provider: Annotated[Provider | None, typer.Option("--provider", "-p", case_sensitive=False)] = None, + output_folder: Annotated[ + str, typer.Option("--output-folder", "-o", help="Folder to store outputs in") + ] = "./outputs", + effective_date: Annotated[ + str | None, typer.Option("--effective-date", help="Effective date for provider queries in YYYY-MM-DD format") + ] = None, + log_level: Annotated[ + LogLevel, typer.Option("--log-level", help="Logging level", case_sensitive=False) + ] = LogLevel.INFO, + no_input: Annotated[ + bool, typer.Option("--no-input", help="Fail instead of prompting for interactive input") + ] = False, + log_dir: Annotated[Path | None, typer.Option("--log-dir", help="Directory to write logs to")] = None, + log_file: Annotated[Path | None, typer.Option("--log-file", help="File path to write logs to")] = None, +): + _configure_interaction(no_input) + _run_raw( + state, + provider, + output_folder, + _parse_effective_date(effective_date), + _log_level_to_int(log_level), + log_dir, + log_file, + ) + + +@urdb_app.callback() +def main_urdb( + ctx: typer.Context, + state: Annotated[ + StateCode | None, typer.Option("--state", "-s", help="Two-letter state abbreviation", case_sensitive=False) + ] = None, + output_folder: Annotated[ + str, typer.Option("--output-folder", "-o", help="Folder to store outputs in") + ] = "./outputs", + year: Annotated[int | None, typer.Option("--year", "-y")] = None, + log_level: Annotated[ + LogLevel, typer.Option("--log-level", help="Logging level", case_sensitive=False) + ] = LogLevel.INFO, + no_input: Annotated[ + bool, typer.Option("--no-input", help="Fail instead of prompting for interactive input") + ] = False, + log_dir: Annotated[Path | None, typer.Option("--log-dir", help="Directory to write logs to")] = None, + log_file: Annotated[Path | None, typer.Option("--log-file", help="File path to write logs to")] = None, + fail_fast: Annotated[ + bool, + typer.Option("--fail-fast", help="Raise conversion errors immediately instead of prompting to continue"), + ] = False, + properties: Annotated[ + list[str] | None, + typer.Option("--property", help="Tariff property override in key=value form; repeat for multiple values"), + ] = None, +): + if ctx.invoked_subcommand is not None: + return + _configure_interaction(no_input) + state_ = state or prompt_state().value + output_folder_ = Path(output_folder) + _ = _configure_command_logging( + "tariff_fetch_urdb", + log_level=_log_level_to_int(log_level), + log_dir=log_dir or (output_folder_ / "logs"), + log_file=log_file, + ) + utility = prompt_utility(state_) + year = prompt_year() if year is None else year + + console.print("Processing [blue]Genability[/]") + _run_cli_command( + lambda: process_genability_urdb( + utility=utility, + output_folder=output_folder_, + year=year, + interactive_errors=not fail_fast, + properties=_parse_property_assignments(properties), + ) + ) + + +@urdb_app.command("ni", help="Convert a specific Arcadia master tariff directly to URDB JSON.") +def urdb_direct( + master_tariff_id: Annotated[int, typer.Argument(help="Arcadia master tariff id to convert")], + year: Annotated[int, typer.Argument(help="Calendar year to convert")], + charge_classes: Annotated[ + list[str] | None, + typer.Option("--charge-class", help="Arcadia charge class to include; repeat to include multiple"), + ] = None, + charge_class_shortcuts: Annotated[ + list[str] | None, + typer.Option( + "-cc", + "--cc", + help=( + "Compact Arcadia charge-class selector. " + "Codes: S=SUPPLY T=TRANSMISSION D=DISTRIBUTION t=TAX " + "C=CONTRACTED U=USER_ADJUSTED A=AFTER_TAX O=OTHER " + "N=NON_BYPASSABLE n=NET_EXCESS" + ), + ), + ] = None, + apply_percentages: Annotated[ + bool, + typer.Option("--apply-percentages/--no-apply-percentages", help="Apply supported percentage rates"), + ] = True, + fail_fast: Annotated[ + bool, + typer.Option("--fail-fast", help="Raise conversion errors immediately instead of prompting to continue"), + ] = False, + properties: Annotated[ + list[str] | None, + typer.Option("--property", help="Tariff property override in key=value form; repeat for multiple values"), + ] = None, + log_level: Annotated[ + LogLevel, typer.Option("--log-level", help="Logging level", case_sensitive=False) + ] = LogLevel.INFO, + no_input: Annotated[ + bool, typer.Option("--no-input", help="Fail instead of prompting for interactive input") + ] = False, + output: Annotated[Path | None, typer.Option("--output", "-o", help="Path to write the converted URDB JSON")] = None, + log_dir: Annotated[Path | None, typer.Option("--log-dir", help="Directory to write logs to")] = None, + log_file: Annotated[Path | None, typer.Option("--log-file", help="File path to write logs to")] = None, + force: Annotated[ + bool, + typer.Option("--force", "-f", help="Overwrite the output file if it already exists"), + ] = False, +): + _configure_interaction(no_input) + _ = load_dotenv() + if output is None: + output = Path("./outputs") + output.mkdir(parents=True, exist_ok=True) + if output.is_dir(): + output = output / f"arcadia_urdb_{master_tariff_id}_{year}.json" + if output.exists() and not force: + console.print(f"[red]Output file already exists: {output}. Pass --force to overwrite it.[/red]") + raise typer.Exit(1) + _ = _configure_command_logging( + "tariff_fetch_urdb", + log_level=_log_level_to_int(log_level), + log_dir=log_dir or (output.parent / "logs"), + log_file=log_file, + ) + scenario_charge_classes = _parse_charge_classes(charge_classes, charge_class_shortcuts) + scenario = Scenario( + master_tariff_id=master_tariff_id, + year=year, + apply_percentages=apply_percentages, + charge_classes=scenario_charge_classes, + properties=_parse_property_assignments(properties), + ) + api = ArcadiaSignalAPI() + result = _run_cli_command(lambda: build_urdb(api, scenario, interactive_errors=not fail_fast)) + _ = output.write_text(json.dumps(result, indent=2)) + + +@gas_app.callback() +def main_gas( + ctx: typer.Context, + state: Annotated[ + StateCode | None, typer.Option("--state", "-s", help="Two-letter state abbreviation", case_sensitive=False) + ] = None, + output_folder: Annotated[ + str, typer.Option("--output-folder", "-o", help="Folder to store outputs in") + ] = "./outputs", + log_level: Annotated[ + LogLevel, typer.Option("--log-level", help="Logging level", case_sensitive=False) + ] = LogLevel.INFO, + no_input: Annotated[ + bool, typer.Option("--no-input", help="Fail instead of prompting for interactive input") + ] = False, + log_dir: Annotated[Path | None, typer.Option("--log-dir", help="Directory to write logs to")] = None, + log_file: Annotated[Path | None, typer.Option("--log-file", help="File path to write logs to")] = None, +): + if ctx.invoked_subcommand is not None: + return + + _configure_interaction(no_input) + state_ = (state or prompt_state()).value + output_folder_ = Path(output_folder) + _ = _configure_command_logging( + "tariff_fetch_gas", + log_level=_log_level_to_int(log_level), + log_dir=log_dir or (output_folder_ / "logs"), + log_file=log_file, + ) + _run_rateacuity_command(lambda: process_rateacuity_gas(output_folder_, state_)) + + +@gas_urdb_app.callback() +def main_gas_urdb( + ctx: typer.Context, + state: Annotated[ + StateCode | None, typer.Option("--state", "-s", help="Two-letter state abbreviation", case_sensitive=False) + ] = None, + output_folder: Annotated[ + str, typer.Option("--output-folder", "-o", help="Folder to store outputs in") + ] = "./outputs", + year: Annotated[int | None, typer.Option("--year", "-y")] = None, + log_level: Annotated[ + LogLevel, typer.Option("--log-level", help="Logging level", case_sensitive=False) + ] = LogLevel.INFO, + no_input: Annotated[ + bool, typer.Option("--no-input", help="Fail instead of prompting for interactive input") + ] = False, + log_dir: Annotated[Path | None, typer.Option("--log-dir", help="Directory to write logs to")] = None, + log_file: Annotated[Path | None, typer.Option("--log-file", help="File path to write logs to")] = None, +): + if ctx.invoked_subcommand is not None: + return + _configure_interaction(no_input) + state_ = (state or prompt_state()).value + output_folder_ = Path(output_folder) + year_ = prompt_year() if year is None else year + _ = _configure_command_logging( + "tariff_fetch_gas_urdb", + log_level=_log_level_to_int(log_level), + log_dir=log_dir or (output_folder_ / "logs"), + log_file=log_file, + ) + _run_rateacuity_command(lambda: process_rateacuity_gas_urdb(output_folder_, state_, year_)) + + +@gas_urdb_app.command("ni", help="Convert gas RateAcuity tariffs to URDB using fuzzy-matched utility and tariff names.") +def main_gas_urdb_ni( + state: Annotated[StateCode, typer.Argument(help="Two-letter state abbreviation")], + utility: Annotated[str, typer.Argument(help="Utility name query to fuzzy-match against RateAcuity choices")], + year: Annotated[int, typer.Option("--year", "-y", help="Calendar year to convert")], + tariffs: Annotated[ + list[str] | None, + typer.Option("--tariff", help="Tariff name query to fuzzy-match; repeat to include multiple tariffs"), + ] = None, + label: Annotated[ + str | None, + typer.Option("--label", help="URDB label override; defaults to an acronym derived from the utility name"), + ] = None, + sector: Annotated[ + Literal["Residential", "Commercial", "Industrial", "Lighting"], + typer.Option("--sector", help="URDB sector"), + ] = "Residential", + servicetype: Annotated[ + Literal["Bundled", "Energy", "Delivery", "Delivery with Standard Offer"], + typer.Option("--servicetype", help="URDB service type"), + ] = "Bundled", + apply_percentages: Annotated[ + bool, + typer.Option("--apply-percentages/--no-apply-percentages", help="Apply supported percentage rows"), + ] = False, + output: Annotated[Path | None, typer.Option("--output", "-o", help="Path to write the converted URDB JSON")] = None, + log_level: Annotated[ + LogLevel, typer.Option("--log-level", help="Logging level", case_sensitive=False) + ] = LogLevel.INFO, + no_input: Annotated[ + bool, typer.Option("--no-input", help="Fail instead of prompting for interactive input") + ] = False, + log_dir: Annotated[Path | None, typer.Option("--log-dir", help="Directory to write logs to")] = None, + log_file: Annotated[Path | None, typer.Option("--log-file", help="File path to write logs to")] = None, + force: Annotated[ + bool, + typer.Option("--force", "-f", help="Overwrite the output file if it already exists"), + ] = False, +): + _run_rateacuity_gas_urdb_ni( + state=state.value, + utility_query=utility, + year=year, + tariffs=tariffs, + label=label, + sector=sector, + servicetype=servicetype, + apply_percentages=apply_percentages, + output=output, + log_level=log_level, + no_input=no_input, + log_dir=log_dir, + log_file=log_file, + force=force, + ) + + +@gas_app.command("ni", help="Fetch gas RateAcuity tariffs by fuzzy-matched state, utility, and tariff names.") +def main_gas_fuzzy( + state: Annotated[StateCode, typer.Argument(help="Two-letter state abbreviation")], + utility: Annotated[str, typer.Argument(help="Utility name query to fuzzy-match against RateAcuity choices")], + tariffs: Annotated[ + list[str] | None, + typer.Option("--tariff", help="Tariff name query to fuzzy-match; repeat to include multiple tariffs"), + ] = None, + output: Annotated[Path | None, typer.Option("--output", "-o", help="Path to write the fetched tariff JSON")] = None, + log_level: Annotated[ + LogLevel, typer.Option("--log-level", help="Logging level", case_sensitive=False) + ] = LogLevel.INFO, + no_input: Annotated[ + bool, typer.Option("--no-input", help="Fail instead of prompting for interactive input") + ] = False, + log_dir: Annotated[Path | None, typer.Option("--log-dir", help="Directory to write logs to")] = None, + log_file: Annotated[Path | None, typer.Option("--log-file", help="File path to write logs to")] = None, + force: Annotated[ + bool, + typer.Option("--force", "-f", help="Overwrite the output file if it already exists"), + ] = False, +): + _run_rateacuity_gas_ni( + state=state.value, + utility_query=utility, + tariffs=tariffs, + output=output, + log_level=log_level, + no_input=no_input, + log_dir=log_dir, + log_file=log_file, + force=force, + ) + + +@ni_app.command("arcadia", help="Fetch a specific Arcadia master tariff as raw JSON.") +def ni_arcadia( + master_tariff_id: Annotated[int, typer.Argument(help="Arcadia master tariff id to fetch")], + effective_date: Annotated[ + str | None, + typer.Argument(help="Effective date in YYYY-MM-DD format; defaults to today if omitted"), + ] = None, + output: Annotated[Path | None, typer.Option("--output", "-o", help="Path to write the fetched tariff JSON")] = None, + log_level: Annotated[ + LogLevel, typer.Option("--log-level", help="Logging level", case_sensitive=False) + ] = LogLevel.INFO, + no_input: Annotated[ + bool, typer.Option("--no-input", help="Fail instead of prompting for interactive input") + ] = False, + log_dir: Annotated[Path | None, typer.Option("--log-dir", help="Directory to write logs to")] = None, + log_file: Annotated[Path | None, typer.Option("--log-file", help="File path to write logs to")] = None, + force: Annotated[ + bool, + typer.Option("--force", "-f", help="Overwrite the output file if it already exists"), + ] = False, +): + _configure_interaction(no_input) + _ = load_dotenv() + effective_on = _parse_effective_date(effective_date) or date.today() + if output is None: + output = Path("./outputs") + output.mkdir(parents=True, exist_ok=True) + if output.is_dir(): + output = output / f"arcadia_{master_tariff_id}_{effective_on.isoformat()}.json" + if output.exists() and not force: + console.print(f"[red]Output file already exists: {output}. Pass --force to overwrite it.[/red]") + raise typer.Exit(1) + + _ = _configure_command_logging( + "tariff_fetch_ni_arcadia", + log_level=_log_level_to_int(log_level), + log_dir=log_dir or (output.parent / "logs"), + log_file=log_file, + ) + results = _run_cli_command( + lambda: _fetch_arcadia_tariffs( + master_tariff_id=master_tariff_id, + effective_on=effective_on, + populate_rates=True, + ) + ) + _ = output.write_bytes(TypeAdapter(list[tariff.TariffExtended]).dump_json(results, indent=2)) + console.print(f"Wrote [blue]{len(results)}[/] records to {output}") + + +@ni_app.command("openei", help="Fetch OpenEI tariffs for a specific utility EIA id as raw JSON.") +def ni_openei( + eia_id: Annotated[int, typer.Argument(help="Utility EIA id to fetch tariffs for")], + sector: Annotated[ + Literal["Residential", "Commercial", "Industrial", "Lighting"], + typer.Argument(help="OpenEI sector to fetch"), + ], + effective_date: Annotated[ + str | None, + typer.Argument(help="Effective date in YYYY-MM-DD format; defaults to today if omitted"), + ] = None, + detail: Annotated[ + Literal["full", "minimal"], typer.Option("--detail", help="OpenEI response detail level") + ] = "full", + labels: Annotated[ + list[str] | None, + typer.Option("--label", help="OpenEI tariff label to include; repeat to include multiple"), + ] = None, + output: Annotated[Path | None, typer.Option("--output", "-o", help="Path to write the fetched tariff JSON")] = None, + log_level: Annotated[ + LogLevel, typer.Option("--log-level", help="Logging level", case_sensitive=False) + ] = LogLevel.INFO, + no_input: Annotated[ + bool, typer.Option("--no-input", help="Fail instead of prompting for interactive input") + ] = False, + log_dir: Annotated[Path | None, typer.Option("--log-dir", help="Directory to write logs to")] = None, + log_file: Annotated[Path | None, typer.Option("--log-file", help="File path to write logs to")] = None, + force: Annotated[ + bool, + typer.Option("--force", "-f", help="Overwrite the output file if it already exists"), + ] = False, +): + _configure_interaction(no_input) + _ = load_dotenv() + effective_on = _parse_effective_date(effective_date) or date.today() + if output is None: + output = Path("./outputs") + output.mkdir(parents=True, exist_ok=True) + if output.is_dir(): + output = output / f"openei_{eia_id}_{sector}_{detail}_{effective_on.isoformat()}.json" + if output.exists() and not force: + console.print(f"[red]Output file already exists: {output}. Pass --force to overwrite it.[/red]") + raise typer.Exit(1) + + _ = _configure_command_logging( + "tariff_fetch_ni_openei", + log_level=_log_level_to_int(log_level), + log_dir=log_dir or (output.parent / "logs"), + log_file=log_file, + ) + results = _run_cli_command( + lambda: _fetch_openei_tariffs( + eia_id=eia_id, + sector=sector, + detail=detail, + effective_on=effective_on, + labels=labels, + ) + ) + _ = output.write_text(json.dumps({"items": results}, indent=2)) + console.print(f"Wrote [blue]{len(results)}[/] items to {output}") + + +@rateacuity_ni_app.command("fuzzy", help="Fetch RateAcuity tariffs by fuzzy-matched state, utility, and tariff names.") +def ni_rateacuity_fuzzy( + state: Annotated[StateCode, typer.Argument(help="Two-letter state abbreviation")], + utility: Annotated[str, typer.Argument(help="Utility name query to fuzzy-match against RateAcuity choices")], + tariffs: Annotated[ + list[str] | None, + typer.Option("--tariff", help="Tariff name query to fuzzy-match; repeat to include multiple tariffs"), + ] = None, + output: Annotated[Path | None, typer.Option("--output", "-o", help="Path to write the fetched tariff JSON")] = None, + log_level: Annotated[ + LogLevel, typer.Option("--log-level", help="Logging level", case_sensitive=False) + ] = LogLevel.INFO, + no_input: Annotated[ + bool, typer.Option("--no-input", help="Fail instead of prompting for interactive input") + ] = False, + log_dir: Annotated[Path | None, typer.Option("--log-dir", help="Directory to write logs to")] = None, + log_file: Annotated[Path | None, typer.Option("--log-file", help="File path to write logs to")] = None, + force: Annotated[ + bool, + typer.Option("--force", "-f", help="Overwrite the output file if it already exists"), + ] = False, +): + _run_rateacuity_ni( + state=state.value, + utility_query=utility, + tariffs=tariffs, + output=output, + log_level=log_level, + no_input=no_input, + log_dir=log_dir, + log_file=log_file, + force=force, + ) + + +@rateacuity_ni_app.command("eia-id", help="Fetch RateAcuity tariffs by utility EIA id via the cached parquet.") +def ni_rateacuity_eia_id( + eia_id: Annotated[int, typer.Argument(help="Utility EIA id to resolve via the cached utilities parquet")], + tariffs: Annotated[ + list[str] | None, + typer.Option("--tariff", help="Tariff name query to fuzzy-match; repeat to include multiple tariffs"), + ] = None, + output: Annotated[Path | None, typer.Option("--output", "-o", help="Path to write the fetched tariff JSON")] = None, + log_level: Annotated[ + LogLevel, typer.Option("--log-level", help="Logging level", case_sensitive=False) + ] = LogLevel.INFO, + no_input: Annotated[ + bool, typer.Option("--no-input", help="Fail instead of prompting for interactive input") + ] = False, + log_dir: Annotated[Path | None, typer.Option("--log-dir", help="Directory to write logs to")] = None, + log_file: Annotated[Path | None, typer.Option("--log-file", help="File path to write logs to")] = None, + force: Annotated[ + bool, + typer.Option("--force", "-f", help="Overwrite the output file if it already exists"), + ] = False, +): + utility_record = _get_utility_by_eia_id(eia_id) + _run_rateacuity_ni( + state=utility_record.state.lower(), + utility_query=utility_record.name, + tariffs=tariffs, + output=output, + log_level=log_level, + no_input=no_input, + log_dir=log_dir, + log_file=log_file, + force=force, + ) + + +@app.command("show-properties", help="Show Arcadia tariff properties for a master tariff.") +def show_properties( + master_tariff_id: Annotated[int, typer.Argument(help="Arcadia master tariff id to inspect")], + effective_date: Annotated[ + str | None, + typer.Argument(help="Effective date in YYYY-MM-DD format; defaults to today if omitted"), + ] = None, + log_level: Annotated[ + LogLevel, typer.Option("--log-level", help="Logging level", case_sensitive=False) + ] = LogLevel.INFO, + no_input: Annotated[ + bool, typer.Option("--no-input", help="Fail instead of prompting for interactive input") + ] = False, + log_dir: Annotated[Path | None, typer.Option("--log-dir", help="Directory to write logs to")] = None, + log_file: Annotated[Path | None, typer.Option("--log-file", help="File path to write logs to")] = None, +): + _configure_interaction(no_input) + _ = load_dotenv() + effective_on = _parse_effective_date(effective_date) or date.today() + _ = _configure_command_logging( + "tariff_fetch_show_properties_arcadia", + log_level=_log_level_to_int(log_level), + log_dir=log_dir or (Path("./outputs") / "logs"), + log_file=log_file, + ) + tariffs = _run_cli_command( + lambda: _fetch_arcadia_tariffs( + master_tariff_id=master_tariff_id, + effective_on=effective_on, + populate_rates=True, + ) + ) + _print_arcadia_properties(tariffs) + + +@cache_app.command("clear", help="Delete the cached EIA utility parquet file.") +def clear_cache(): + if not UTILITY_CACHE_PATH.exists(): + console.print(f"No cached utilities parquet found at [blue]{UTILITY_CACHE_PATH}[/]") + return + + UTILITY_CACHE_PATH.unlink() + console.print(f"Cleared cached utilities parquet at [blue]{UTILITY_CACHE_PATH}[/]") + + +@cache_app.command("location", help="Show the cached EIA utility parquet file path.") +def cache_location(): + console.print(f"Utility parquet cache path: [blue]{UTILITY_CACHE_PATH}[/]") + + +def main_cli(): + app() + + +def _run_raw( + state: StateCode | None, + provider: Provider | None, + output_folder: str, + effective_date: date | None, + log_level: int, + log_dir: Path | None, + log_file: Path | None, +): + state_ = state or prompt_state().value + provider = provider or prompt_provider() + output_folder_ = Path(output_folder) + _ = _configure_command_logging( + "tariff_fetch", + log_level=log_level, + log_dir=log_dir or (output_folder_ / "logs"), + log_file=log_file, + ) + utility = prompt_utility(state_) + + match provider: + case Provider.GENABILITY: + console.print("Processing [blue]Genability[/]") + _run_cli_command( + lambda: process_genability(utility=utility, output_folder=output_folder_, effective_on=effective_date) + ) + case Provider.OPENEI: + console.print("Processing [blue]OpenEI[/]") + _run_cli_command(lambda: process_openei(utility, output_folder_, effective_on=effective_date)) + case Provider.RATEACUITY: + _run_rateacuity_command(lambda: process_rateacuity(output_folder_, state_, utility)) + + +def _configure_logging( + suffix: str, + *, + log_level: int, + log_dir: Path | None = None, + log_file: Path | None = None, +) -> Path: + if log_dir is not None and log_file is not None: + raise typer.BadParameter("Use either --log-dir or --log-file, not both.") + + if log_file is None: + log_dir = log_dir or Path("./outputs/logs") + log_dir.mkdir(parents=True, exist_ok=True) + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + path = log_dir / f"{suffix}_{timestamp}.log" + else: + log_file.parent.mkdir(parents=True, exist_ok=True) + path = log_file + + rich_handler = RichHandler(rich_tracebacks=True) + rich_handler.setLevel(log_level) + rich_handler.setFormatter(logging.Formatter("%(message)s")) + + file_handler = logging.FileHandler(path, encoding="utf-8") + file_handler.setLevel(log_level) + file_handler.setFormatter(logging.Formatter(LOG_FORMAT)) + + logging.basicConfig(level=log_level, handlers=[rich_handler, file_handler], force=True) + _configure_noisy_loggers(log_level) + return path + + +def _configure_command_logging( + suffix: str, + *, + log_level: int, + log_dir: Path | None = None, + log_file: Path | None = None, +) -> Path: + log_path = _configure_logging(suffix, log_level=log_level, log_dir=log_dir, log_file=log_file) + console.print(f"Logging to [blue]{log_path}[/]") + return log_path + + +def _configure_interaction(no_input: bool) -> None: + q.set_no_input(no_input) + + +def _run_cli_command(command: Callable[[], _T]) -> _T: try: - _ = date(int(value), 1, 1) - except (TypeError, ValueError): - return False - return True + return command() + except typer.Exit as e: + _handle_expected_exit(e) + raise AssertionError("unreachable") from None + except Exception as e: + logging.getLogger(__name__).exception(e) + raise typer.Exit(1) from e -def prompt_state() -> StateCode: - choice = Prompt.ask( - "Enter two-letter state abbreviation", - choices=[state.value for state in StateCode], - show_choices=False, - case_sensitive=False, +def _run_rateacuity_command(command: Callable[[], _T]) -> _T: + try: + return command() + except typer.Exit as e: + _handle_expected_exit(e) + raise AssertionError("unreachable") from None + except AuthorizationError: + _print_authorization_failed() + raise typer.Exit(1) from None + except Exception as e: + logging.getLogger(__name__).exception(e) + raise typer.Exit(1) from e + + +def _print_authorization_failed() -> None: + console.print("Authorization failed") + console.print( + "Check if credentials provided via [b]RATEACUITY_USERNAME[/] and [b]RATEACUITY_PASSWORD[/] environment variables are correct" ) - return StateCode(choice.lower()) -def prompt_providers() -> list[Provider]: - return cast( - list[Provider], - questionary.checkbox( - message="Select providers", - choices=[questionary.Choice(title=_.value, value=_) for _ in Provider], - validate=lambda x: True if x else "Select at least one provider", - ).ask(), +def _parse_effective_date(value: str | None) -> date | None: + if value is None: + return None + try: + return date.fromisoformat(value) + except ValueError as exc: + raise typer.BadParameter("Effective date must be in YYYY-MM-DD format.") from exc + + +def _log_level_to_int(value: LogLevel) -> int: + level = getattr(logging, value.value, None) + if not isinstance(level, int): + raise typer.BadParameter(f"Unsupported log level: {value.value}") + return level + + +def _configure_noisy_loggers(log_level: int) -> None: + # Browser automation pulls in urllib3/http chatter and asyncio selector noise. + noisy_logger_names = ( + "selenium", + "urllib3", + "urllib3.connectionpool", + "asyncio", ) + noisy_level = max(log_level, logging.INFO) + for logger_name in noisy_logger_names: + logging.getLogger(logger_name).setLevel(noisy_level) + + +def _handle_expected_exit(exc: typer.Exit) -> None: + if exc.exit_code not in (None, 0): + console.print("[yellow]Cancelled by user[/]") + raise exc + + +def prompt_provider() -> Provider: + return q.select( + message="Select provider", + choices=[q.Choice(title=provider.value, value=provider) for provider in Provider], + ).ask_or_exit() + + +class _UtilityLookup(NamedTuple): + eia_id: int + name: str + state: str def prompt_utility(state: str) -> Utility: with console.status("Fetching utilities..."): yearly_sales_df = ( - pl.read_parquet(CORE_EIA861_YEARLY_SALES_HTTPS) # pyright: ignore[reportUnknownMemberType] + pl.read_parquet(_get_cached_utility_sales_parquet()) # pyright: ignore[reportUnknownMemberType] .filter(pl.col("state") == state.upper()) .filter(pl.col("report_date") == pl.col("report_date").max().over("utility_id_eia")) # pyright: ignore[reportUnknownMemberType] .filter(pl.col("entity_type").is_in(ENTITY_TYPES_SORTORDER)) @@ -96,137 +910,434 @@ def fmt_number(value: float | int | None) -> str: return f"{value:,.0f}" utility_name_header = "Utility Name" + eia_id_header = "EIA ID" entity_type_header = "Entity Type" sales_header = "Sales (MWh)" revenue_header = "Revenue ($)" customers_header = "Customers" largest_utility_name = max(len(utility_name_header), *(len(row["utility_name"]) for row in rows)) # pyright: ignore[reportAny] + largest_eia_id = max(len(eia_id_header), *(len(str(row["utility_id_eia"])) for row in rows)) # pyright: ignore[reportAny] largest_entity_type = max(len(entity_type_header), *(len(row["entity_type"][:18]) for row in rows)) # pyright: ignore[reportAny] largest_sales_col = max(len(sales_header), *(len(fmt_number(row["sales_mwh"])) for row in rows)) # pyright: ignore[reportAny] largest_revenue_col = max(len(revenue_header), *(len(fmt_number(row["sales_revenue"])) for row in rows)) # pyright: ignore[reportAny] largest_customers_col = max(len(customers_header), *(len(fmt_number(row["customers"])) for row in rows)) # pyright: ignore[reportAny] header_str_utility_name = utility_name_header.ljust(largest_utility_name) + header_str_eia_id = eia_id_header.ljust(largest_eia_id) header_str_entity_type = entity_type_header.ljust(largest_entity_type) header_str_sales = sales_header.ljust(largest_sales_col) header_str_revenue = revenue_header.ljust(largest_revenue_col) header_str_customers = customers_header.ljust(largest_customers_col) - header_str = f"{header_str_utility_name} | {header_str_entity_type} | {header_str_sales} | {header_str_revenue} | {header_str_customers}" - separator = questionary.Separator(line="-" * len(header_str)) + header_str = ( + f"{header_str_utility_name} | {header_str_eia_id} | {header_str_entity_type} | " + f"{header_str_sales} | {header_str_revenue} | {header_str_customers}" + ) + separator = q.Separator(line="-" * len(header_str)) - header = questionary.Choice( + header = q.Choice[Utility | None]( title=header_str, - value=0, + value=None, ) - def build_choice(row: dict[str, str | int | float | None]) -> questionary.Choice: + def build_choice(row: dict[str, str | int | float | None]) -> q.Choice[Utility | None]: name_col = cast(str, row["utility_name"]).ljust(largest_utility_name) + eia_id_col = str(cast(int, row["utility_id_eia"])).ljust(largest_eia_id) entity_type = (cast(str, row["entity_type"]) or "-")[:18].ljust(largest_entity_type) sales_col = fmt_number(cast(float, row["sales_mwh"])).ljust(largest_sales_col) revenue_col = fmt_number(cast(float, row["sales_revenue"])).ljust(largest_revenue_col) customers_col = fmt_number(cast(float, row["customers"])).ljust(largest_customers_col) - title = f"{name_col} | {entity_type} | {sales_col} | {revenue_col} | {customers_col}" - return questionary.Choice( + title = f"{name_col} | {eia_id_col} | {entity_type} | {sales_col} | {revenue_col} | {customers_col}" + return q.Choice( title=title, value=Utility(eia_id=cast(int, row["utility_id_eia"]), name=cast(str, row["utility_name"])), ) - result = 0 - while result == 0: - result = cast( - Utility, - questionary.select( - message="Select a utility", - choices=[header, separator, *[build_choice(row) for row in rows]], - use_search_filter=True, - use_jk_keys=False, - use_shortcuts=False, - ).ask(), + result: Utility | None = None + while result is None: + result = q.select( + message="Select a utility", + choices=[header, separator, *[build_choice(row) for row in rows]], + use_search_filter=True, + use_jk_keys=False, + use_shortcuts=False, + ).ask_or_exit() + return result + + +def _get_utility_by_eia_id(eia_id: int) -> _UtilityLookup: + with console.status("Resolving utility from cached parquet..."): + rows = ( + pl.read_parquet(_get_cached_utility_sales_parquet()) # pyright: ignore[reportUnknownMemberType] + .filter(pl.col("utility_id_eia") == eia_id) + .filter(pl.col("report_date") == pl.col("report_date").max().over("utility_id_eia")) # pyright: ignore[reportUnknownMemberType] + .select("utility_id_eia", "utility_name_eia", "state") + .unique() + .iter_rows(named=True) + ) + row = next(rows, None) + + if row is None: + raise typer.BadParameter( + f"No utility with EIA ID {eia_id} was found in the cached parquet.", param_hint="--eia-id" + ) + + return _UtilityLookup( + eia_id=cast(int, row["utility_id_eia"]), + name=cast(str, row["utility_name_eia"]), + state=cast(str, row["state"]), + ) + + +def prompt_year() -> int: + result = q.text("Enter year", default=str(date.today().year - 1), validate=_is_valid_year).ask_or_exit() + return int(result) + + +def prompt_state() -> StateCode: + return q.select( + message="Select state", + choices=[q.Choice(title=state.value.upper(), value=state) for state in StateCode], + use_search_filter=True, + use_jk_keys=False, + use_shortcuts=False, + ).ask_or_exit() + + +def _parse_charge_classes( + charge_classes: list[str] | None, charge_class_shortcuts: list[str] | None = None +) -> set[RateChargeClass]: + if charge_classes is None and charge_class_shortcuts is None: + return set(ALL_CHARGE_CLASSES) + + normalized = [charge_class.strip().upper() for charge_class in (charge_classes or [])] + invalid = sorted(set(normalized) - set(ALL_CHARGE_CLASSES)) + shortcut_invalid = sorted( + {code for shortcut in charge_class_shortcuts or [] for code in shortcut if code not in CHARGE_CLASS_SHORTCUTS} + ) + if shortcut_invalid: + allowed = "".join(CHARGE_CLASS_SHORTCUTS) + console.print(f"[red]Invalid --cc codes:[/] {', '.join(shortcut_invalid)}") + console.print(f"Allowed values: {allowed}") + raise typer.Exit(code=1) + normalized.extend(CHARGE_CLASS_SHORTCUTS[code] for shortcut in charge_class_shortcuts or [] for code in shortcut) + if invalid: + allowed = ", ".join(ALL_CHARGE_CLASSES) + console.print(f"[red]Invalid charge classes:[/] {', '.join(invalid)}") + console.print(f"Allowed values: {allowed}") + raise typer.Exit(code=1) + return {cast(RateChargeClass, charge_class) for charge_class in normalized} + + +def _run_rateacuity_ni( + *, + state: str, + utility_query: str, + tariffs: list[str] | None, + output: Path | None, + log_level: LogLevel, + no_input: bool, + log_dir: Path | None, + log_file: Path | None, + force: bool, +) -> None: + _configure_interaction(no_input) + if not tariffs: + raise typer.BadParameter("Pass at least one --tariff value.", param_hint="--tariff") + if output is None: + output = Path("./outputs") + output.mkdir(parents=True, exist_ok=True) + elif output.exists() and output.is_file() and not force: + console.print(f"[red]Output file already exists: {output}. Pass --force to overwrite it.[/red]") + raise typer.Exit(1) + + _ = _configure_command_logging( + "tariff_fetch_ni_rateacuity", + log_level=_log_level_to_int(log_level), + log_dir=log_dir or ((output if output.is_dir() else output.parent) / "logs"), + log_file=log_file, + ) + selected_utility, results = _run_rateacuity_command( + lambda: fetch_rateacuity_tariffs(state=state, utility_query=utility_query, tariff_queries=tariffs) + ) + if output.is_dir(): + output = output / f"{sanitize_filename(f'rateacuity_{selected_utility}')}.json" + if output.exists() and not force: + console.print(f"[red]Output file already exists: {output}. Pass --force to overwrite it.[/red]") + raise typer.Exit(1) + output.parent.mkdir(parents=True, exist_ok=True) + _ = output.write_text(json.dumps(results, indent=2)) + console.print(f"Wrote [blue]{len(results)}[/] records to {output}") + + +def _run_rateacuity_gas_ni( + *, + state: str, + utility_query: str, + tariffs: list[str] | None, + output: Path | None, + log_level: LogLevel, + no_input: bool, + log_dir: Path | None, + log_file: Path | None, + force: bool, +) -> None: + _configure_interaction(no_input) + if not tariffs: + raise typer.BadParameter("Pass at least one --tariff value.", param_hint="--tariff") + if output is None: + output = Path("./outputs") + output.mkdir(parents=True, exist_ok=True) + elif output.exists() and output.is_file() and not force: + console.print(f"[red]Output file already exists: {output}. Pass --force to overwrite it.[/red]") + raise typer.Exit(1) + + _ = _configure_command_logging( + "tariff_fetch_gas_fuzzy", + log_level=_log_level_to_int(log_level), + log_dir=log_dir or ((output if output.is_dir() else output.parent) / "logs"), + log_file=log_file, + ) + selected_utility, results = _run_rateacuity_command( + lambda: fetch_rateacuity_gas_tariffs(state=state, utility_query=utility_query, tariff_queries=tariffs) + ) + if output.is_dir(): + output = output / f"{sanitize_filename(f'gas_rateacuity_{selected_utility}')}.json" + if output.exists() and not force: + console.print(f"[red]Output file already exists: {output}. Pass --force to overwrite it.[/red]") + raise typer.Exit(1) + output.parent.mkdir(parents=True, exist_ok=True) + _ = output.write_text(json.dumps(results, indent=2)) + console.print(f"Wrote [blue]{len(results)}[/] records to {output}") + + +def _run_rateacuity_gas_urdb_ni( + *, + state: str, + utility_query: str, + year: int, + tariffs: list[str] | None, + label: str | None, + sector: Literal["Residential", "Commercial", "Industrial", "Lighting"], + servicetype: Literal["Bundled", "Energy", "Delivery", "Delivery with Standard Offer"], + apply_percentages: bool, + output: Path | None, + log_level: LogLevel, + no_input: bool, + log_dir: Path | None, + log_file: Path | None, + force: bool, +) -> None: + _configure_interaction(no_input) + if not tariffs: + raise typer.BadParameter("Pass at least one --tariff value.", param_hint="--tariff") + if output is None: + output = Path("./outputs") + output.mkdir(parents=True, exist_ok=True) + elif output.exists() and output.is_file() and not force: + console.print(f"[red]Output file already exists: {output}. Pass --force to overwrite it.[/red]") + raise typer.Exit(1) + + _ = _configure_command_logging( + "tariff_fetch_gas_urdb_ni", + log_level=_log_level_to_int(log_level), + log_dir=log_dir or ((output if output.is_dir() else output.parent) / "logs"), + log_file=log_file, + ) + selected_utility, result = _run_rateacuity_command( + lambda: fetch_rateacuity_gas_urdb_rates( + state=state, + utility_query=utility_query, + tariff_queries=tariffs, + year=year, + apply_percentages=apply_percentages, + label=label, + sector=sector, + servicetype=servicetype, ) + ) + if output.is_dir(): + output = output / f"{sanitize_filename(f'rateacuity_{selected_utility}.urdb.{year}.')}.json" + if output.exists() and not force: + console.print(f"[red]Output file already exists: {output}. Pass --force to overwrite it.[/red]") + raise typer.Exit(1) + output.parent.mkdir(parents=True, exist_ok=True) + _ = output.write_text(json.dumps({"items": result}, indent=2)) + console.print(f"Wrote [blue]{len(result)}[/] items to {output}") + + +def _parse_property_assignments(values: list[str] | None) -> dict[str, ScenarioPropertyValue]: + if values is None: + return {} + + result: dict[str, ScenarioPropertyValue] = {} + for value in values: + if "=" not in value: + raise typer.BadParameter("Property overrides must use key=value format.", param_hint="--property") + key, raw_value = value.split("=", 1) + key = key.strip() + if not key: + raise typer.BadParameter("Property overrides must include a property key.", param_hint="--property") + existing = result.get(key) + if existing is None: + result[key] = raw_value + elif isinstance(existing, list): + existing.append(raw_value) + elif isinstance(existing, str): + result[key] = [existing, raw_value] + else: + raise typer.BadParameter( + f"Property override for {key} was parsed into an unexpected type.", param_hint="--property" + ) return result -def main( - state: Annotated[ - StateCode | None, typer.Option("--state", "-s", help="Two-letter state abbreviation", case_sensitive=False) - ] = None, - providers: Annotated[list[Provider] | None, typer.Option("--providers", "-p", case_sensitive=False)] = None, - output_folder: Annotated[ - str, typer.Option("--output-folder", "-o", help="Folder to store outputs in") - ] = "./outputs", - urdb: bool = False, -): - logging.basicConfig(level=logging.DEBUG) - # print(pl.read_parquet(CoreEIA861_ASSN_UTILITY.https)) - state_ = state or prompt_state().value - providers_ = providers or prompt_providers() - output_folder_ = Path(output_folder) - utility = prompt_utility(state_) +def _fetch_arcadia_tariffs( + *, + master_tariff_id: int, + effective_on: date, + populate_rates: bool, +) -> list[tariff.TariffExtended]: + api = ArcadiaSignalAPI() + return list( + api.tariffs.iter_pages( + fields="ext", + master_tariff_id=master_tariff_id, + effective_on=effective_on, + populate_properties=True, + populate_rates=populate_rates, + ) + ) - if urdb: - year = prompt_year() - if Provider.GENABILITY in providers_: - console.print("Processing [blue]Genability[/]") - try: - process_genability_urdb(utility=utility, output_folder=output_folder_, year=year) - except HTTPError as e: - if e.response.status_code == 401: - console.print("Authorization failed") - console.print( - "Check if credentials set via [b]ARCADIA_APP_ID[/] and [b]ARCADIA_APP_KEY[/] environment variables are correct" - ) - else: - raise e from None - if Provider.OPENEI in providers_: - console.print("Processing [blue]OpenEI[/]") - try: - process_openei(utility, output_folder_) - except HTTPError as e: - if e.response.status_code == 403: - console.print("Authorization failed") - console.print("Check if [b]OPENEI_API_KEY[/] environment variable is correct") - else: - raise e from None - if Provider.RATEACUITY in providers_: - console.print("Cannot convert RateAcuity data to URDB") - else: - if Provider.GENABILITY in providers_: - console.print("Processing [blue]Genability[/]") - try: - process_genability(utility=utility, output_folder=output_folder_) - except HTTPError as e: - if e.response.status_code == 401: - console.print("Authorization failed") - console.print( - "Check if credentials set via [b]ARCADIA_APP_ID[/] and [b]ARCADIA_APP_KEY[/] environment variables are correct" - ) - else: - raise e from None - if Provider.OPENEI in providers_: - console.print("Processing [blue]OpenEI[/]") - try: - process_openei(utility, output_folder_) - except HTTPError as e: - if e.response.status_code == 403: - console.print("Authorization failed") - console.print("Check if [b]OPENEI_API_KEY[/] environment variable is correct") - else: - raise e from None - if Provider.RATEACUITY in providers_: - console.print("Processing [blue]RateAcuity[/]") - try: - process_rateacuity(output_folder_, state_, utility) - except AuthorizationError: - console.print("Authorization failed") - console.print( - "Check if credentials provided via [b]RATEACUITY_USERNAME[/] and [b]RATEACUITY_PASSWORD[/] environment variables are correct" - ) +def _fetch_openei_tariffs( + *, + eia_id: int, + sector: UtilityRateSector, + detail: str, + effective_on: date, + labels: list[str] | None = None, +) -> list[UtilityRatesResponseItem]: + api_key = os.getenv("OPENEI_API_KEY") + if not api_key: + raise ValueError("API Key is not set (via OPENEI_API_KEY variable)") + with console.status("Fetching rates..."): + results = list( + iter_utility_rates( + api_key, + effective_on_date=datetime.combine(effective_on, datetime.min.time(), tzinfo=UTC), + sector=sector, + detail=cast(Literal["full", "minimal"], detail), + eia=eia_id, + ) + ) + if labels is None: + return results + allowed = set(labels) + return [item for item in results if item["label"] in allowed] -def main_cli(): - typer.run(main) +def _print_arcadia_properties(tariffs: list[tariff.TariffExtended]) -> None: + property_rows = _collect_arcadia_property_rows(tariffs) + + if not property_rows: + console.print("[yellow]No Arcadia properties found for this tariff.[/yellow]") + return + + table = Table(title="Arcadia tariff properties") + table.add_column("Key") + table.add_column("Name") + table.add_column("Type") + table.add_column("Description") + table.add_column("Choices") + for key_name, (display_name, data_type, description, choices) in sorted(property_rows.items()): + table.add_row(key_name, display_name, data_type, description, choices) + console.print(table) + + +def _collect_arcadia_property_rows(tariffs: list[tariff.TariffExtended]) -> dict[str, tuple[str, str, str, str]]: + property_rows: dict[str, dict[str, str | dict[str, str]]] = {} + + for tariff_ in tariffs: + for prop in tariff_.get("properties", []): + key_name = prop["key_name"] + if key_name == "chargeClass": + continue + row = property_rows.setdefault( + key_name, + { + "display_name": prop["display_name"], + "data_type": prop["data_type"], + "description": prop["description"], + "choices_by_value": {}, + }, + ) + choices_by_value = cast(dict[str, str], row["choices_by_value"]) + for choice in prop.get("choices", []): + _ = choices_by_value.setdefault(choice["value"], choice["display_value"]) + + return {key_name: _format_arcadia_property_row(row) for key_name, row in property_rows.items()} + + +def _format_arcadia_property_row(row: dict[str, str | dict[str, str]]) -> tuple[str, str, str, str]: + choices_by_value = cast(dict[str, str], row["choices_by_value"]) + choices = ", ".join(f"{display_value}={value}" for value, display_value in sorted(choices_by_value.items())) + return ( + cast(str, row["display_name"]), + cast(str, row["data_type"]), + cast(str, row["description"]), + choices, + ) + + +def _get_cached_utility_sales_parquet(now: datetime | None = None) -> Path: + logger = logging.getLogger(__name__) + current_time = now or datetime.now() + UTILITY_CACHE_DIR.mkdir(parents=True, exist_ok=True) + + if _is_fresh_cache(UTILITY_CACHE_PATH, current_time): + logger.debug("Using cached utility parquet at %s", UTILITY_CACHE_PATH) + return UTILITY_CACHE_PATH + + try: + _download_utility_sales_parquet(UTILITY_CACHE_PATH) + except Exception: + if UTILITY_CACHE_PATH.exists(): + logger.warning( + "Failed to refresh utility parquet cache; falling back to stale cache at %s", + UTILITY_CACHE_PATH, + exc_info=True, + ) + return UTILITY_CACHE_PATH + raise + + logger.debug("Refreshed utility parquet cache at %s", UTILITY_CACHE_PATH) + return UTILITY_CACHE_PATH + + +def _is_fresh_cache(path: Path, now: datetime) -> bool: + if not path.exists(): + return False + age_seconds = now.timestamp() - path.stat().st_mtime + return age_seconds < UTILITY_CACHE_TTL_SECONDS + + +def _download_utility_sales_parquet(destination: Path) -> None: + with ( + cast(BinaryIO, urlopen(CORE_EIA861_YEARLY_SALES_HTTPS)) as response, + NamedTemporaryFile(dir=destination.parent, delete=False) as temporary_file, + ): + shutil.copyfileobj(response, temporary_file) + temp_path = Path(temporary_file.name) + _ = temp_path.replace(destination) + + +def _is_valid_year(value: str) -> bool: + try: + _ = date(int(value), 1, 1) + except (TypeError, ValueError): + return False + return True if __name__ == "__main__": diff --git a/tariff_fetch/cli_arcadia_urdb.py b/tariff_fetch/cli_arcadia_urdb.py deleted file mode 100644 index bf2f227..0000000 --- a/tariff_fetch/cli_arcadia_urdb.py +++ /dev/null @@ -1,107 +0,0 @@ -"""Direct CLI for converting one Arcadia master tariff to URDB.""" - -import json -import logging -import os -from pathlib import Path -from typing import Annotated, cast, get_args - -import typer -from dotenv import load_dotenv -from rich.logging import RichHandler - -from tariff_fetch.arcadia.api import ArcadiaSignalAPI -from tariff_fetch.arcadia.schema.common import RateChargeClass -from tariff_fetch.urdb.arcadia.build import build_urdb -from tariff_fetch.urdb.arcadia.scenario import Scenario - -from ._cli import console - -FORMAT = "%(message)s" -logging.basicConfig(level="NOTSET", format=FORMAT, datefmt="[%X]", handlers=[RichHandler()]) - -DEFAULT_CHARGE_CLASSES: tuple[RateChargeClass, ...] = ( - "DISTRIBUTION", - "SUPPLY", - "TRANSMISSION", - "OTHER", - "CONTRACTED", -) -ALL_CHARGE_CLASSES = cast(tuple[RateChargeClass, ...], get_args(RateChargeClass)) - - -def main( - master_tariff_id: Annotated[int, typer.Argument(help="Arcadia master tariff id to convert")], - year: Annotated[int, typer.Argument(help="Calendar year to convert")], - output: Annotated[ - Path | None, - typer.Option("--output", "-o", help="Path to write the converted URDB JSON"), - ] = None, - apply_percentages: Annotated[ - bool, - typer.Option("--apply-percentages/--no-apply-percentages", help="Apply supported percentage rates"), - ] = False, - charge_classes: Annotated[ - list[str] | None, - typer.Option("--charge-class", help="Arcadia charge class to include; repeat to include multiple"), - ] = None, - force: Annotated[ - bool, - typer.Option("--force", "-f", help="Overwrite the output file if it already exists"), - ] = False, -) -> None: - """Convert one Arcadia master tariff to a URDB JSON file.""" - - _ = load_dotenv() - if not os.getenv("ARCADIA_APP_ID"): - console.print("[b]ARCADIA_APP_ID[/] environment variable is not set.") - if not os.getenv("ARCADIA_APP_KEY"): - console.print("[b]ARCADIA_APP_KEY[/] environment variable is not set.") - if not (os.getenv("ARCADIA_APP_ID") and os.getenv("ARCADIA_APP_KEY")): - raise typer.Exit(code=1) - - output_path = output or Path(f"./outputs/arcadia_urdb_{master_tariff_id}_{year}.json") - if output_path.exists() and not force: - console.print(f"[red]Output file already exists:[/] {output_path}") - console.print("Pass [b]--force[/] to overwrite it.") - raise typer.Exit(code=1) - - scenario_charge_classes = _parse_charge_classes(charge_classes) - scenario = Scenario( - master_tariff_id=master_tariff_id, - year=year, - apply_percentages=apply_percentages, - charge_classes=scenario_charge_classes, - ) - api = ArcadiaSignalAPI() - - console.print("Converting Arcadia tariff to URDB...") - result = build_urdb(api, scenario) - - output_path.parent.mkdir(parents=True, exist_ok=True) - _ = output_path.write_text(json.dumps(result, indent=2)) - console.print(f"Wrote URDB tariff to [blue]{output_path}[/]") - - -def main_cli() -> None: - """Run the direct Arcadia-to-URDB CLI.""" - - typer.run(main) - - -def _parse_charge_classes(charge_classes: list[str] | None) -> set[RateChargeClass]: - if charge_classes is None: - return set(DEFAULT_CHARGE_CLASSES) - - normalized = [charge_class.strip().upper() for charge_class in charge_classes] - invalid = sorted(set(normalized) - set(ALL_CHARGE_CLASSES)) - if invalid: - allowed = ", ".join(ALL_CHARGE_CLASSES) - console.print(f"[red]Invalid charge classes:[/] {', '.join(invalid)}") - console.print(f"Allowed values: {allowed}") - raise typer.Exit(code=1) - return {cast(RateChargeClass, charge_class) for charge_class in normalized} - - -if __name__ == "__main__": - main_cli() diff --git a/tariff_fetch/cli_gas.py b/tariff_fetch/cli_gas.py deleted file mode 100644 index a12284c..0000000 --- a/tariff_fetch/cli_gas.py +++ /dev/null @@ -1,72 +0,0 @@ -from datetime import date -from pathlib import Path -from typing import Annotated, cast - -import questionary -import rich -import typer -from rich.prompt import Prompt - -from tariff_fetch._cli.rateacuity import process_rateacuity_gas -from tariff_fetch._cli.rateacuity_gas_urdb import process_rateacuity_gas_urdb -from tariff_fetch.rateacuity.base import AuthorizationError - -from ._cli.types import StateCode - - -def prompt_state() -> StateCode: - choice = Prompt.ask( - "Enter two-letter state abbreviation", - choices=[state.value for state in StateCode], - show_choices=False, - case_sensitive=False, - ) - return StateCode(choice.lower()) - - -def prompt_year() -> int: - result = cast( - str, questionary.text("Enter year", default=str(date.today().year - 1), validate=_is_valid_year).ask() - ) - return int(result) - - -def _is_valid_year(value: str) -> bool: - try: - _ = date(int(value), 1, 1) - except (TypeError, ValueError): - return False - return True - - -def main( - state: Annotated[ - StateCode | None, typer.Option("--state", "-s", help="Two-letter state abbreviation", case_sensitive=False) - ] = None, - output_folder: Annotated[ - str, typer.Option("--output-folder", "-o", help="Folder to store outputs in") - ] = "./outputs", - urdb: bool = False, -): - # print(pl.read_parquet(CoreEIA861_ASSN_UTILITY.https)) - state_ = (state or prompt_state()).value - output_folder_ = Path(output_folder) - try: - if urdb: - year = prompt_year() - process_rateacuity_gas_urdb(output_folder_, state_, year) - else: - process_rateacuity_gas(output_folder_, state_) - except AuthorizationError: - rich.print("Authorization failed") - rich.print( - "Check if credentials provided via [b]RATEACUITY_USERNAME[/] and [b]RATEACUITY_PASSWORD[/] environment variables are correct" - ) - - -def main_cli(): - typer.run(main) - - -if __name__ == "__main__": - main_cli() diff --git a/tariff_fetch/questionary_typed.py b/tariff_fetch/questionary_typed.py new file mode 100644 index 0000000..c4e51ab --- /dev/null +++ b/tariff_fetch/questionary_typed.py @@ -0,0 +1,261 @@ +from __future__ import annotations + +from collections.abc import Callable, Sequence +from dataclasses import dataclass +from typing import Generic, Protocol, TypeVar, cast, overload + +import questionary +import typer + +T = TypeVar("T") +_ValidateStr = Callable[[str], bool | str] +_ValidateObjects = Callable[[list[object]], bool | str] + + +class _Question(Protocol): + def ask(self) -> object | None: ... + + +class _SelectFactory(Protocol): + def __call__( + self, + message: str, + choices: Sequence[object], + default: object | None = None, + use_shortcuts: bool = False, + use_arrow_keys: bool = True, + use_jk_keys: bool = True, + use_search_filter: bool = False, + show_description: bool = True, + ) -> _Question: ... + + +class _CheckboxFactory(Protocol): + def __call__( + self, + message: str, + choices: Sequence[object], + validate: _ValidateObjects = ..., + use_jk_keys: bool = True, + use_search_filter: bool = False, + show_description: bool = True, + ) -> _Question: ... + + +class _TextFactory(Protocol): + def __call__( + self, + message: str, + default: str = "", + validate: _ValidateStr = ..., + ) -> _Question: ... + + +class _ConfirmFactory(Protocol): + def __call__(self, message: str, default: bool = True, auto_enter: bool = True) -> _Question: ... + + +class _PathFactory(Protocol): + def __call__( + self, + message: str, + default: str = "", + validate: _ValidateStr = ..., + file_filter: Callable[[str], bool] | None = None, + ) -> _Question: ... + + +_select_impl = cast(_SelectFactory, questionary.select) +_checkbox_impl = cast(_CheckboxFactory, questionary.checkbox) +_text_impl = cast(_TextFactory, questionary.text) +_confirm_impl = cast(_ConfirmFactory, questionary.confirm) +_path_impl = cast(_PathFactory, questionary.path) +_no_input = False + + +def _validate_str_always(_: str) -> bool | str: + return True + + +def _validate_objects_always(_: list[object]) -> bool | str: + return True + + +def set_no_input(enabled: bool) -> None: + global _no_input + _no_input = enabled + + +def is_no_input() -> bool: + return _no_input + + +@dataclass(frozen=True) +class Choice(Generic[T]): + title: str + value: T | None = None + disabled: str | None = None + checked: bool | None = False + shortcut_key: str | bool | None = True + description: str | None = None + + +@dataclass(frozen=True) +class Separator: + line: str | None = None + + +class Prompt(Generic[T]): + _question: _Question + _message: str + + def __init__(self, question: _Question, *, message: str) -> None: + self._question = question + self._message = message + + def ask(self) -> T | None: + if _no_input: + typer.echo(f"Prompt requires interactive input but --no-input was set: {self._message}", err=True) + raise typer.Exit(code=1) + return cast(T | None, self._question.ask()) + + def ask_or_exit(self, code: int = 1) -> T: + result = self.ask() + if result is None: + raise typer.Exit(code=code) + return result + + +def _convert_choice(choice: str | Choice[T] | Separator) -> object: + if isinstance(choice, Choice): + return questionary.Choice( + title=choice.title, + value=choice.value, + disabled=choice.disabled, + checked=choice.checked, + shortcut_key=choice.shortcut_key, + description=choice.description, + ) + if isinstance(choice, Separator): + return questionary.Separator(line=choice.line) + return choice + + +@overload +def select( + message: str, + choices: Sequence[str | Separator], + *, + default: str | None = None, + use_shortcuts: bool = False, + use_arrow_keys: bool = True, + use_jk_keys: bool = True, + use_search_filter: bool = False, + show_description: bool = True, +) -> Prompt[str]: ... + + +@overload +def select( + message: str, + choices: Sequence[Choice[T] | Separator], + *, + default: None = None, + use_shortcuts: bool = False, + use_arrow_keys: bool = True, + use_jk_keys: bool = True, + use_search_filter: bool = False, + show_description: bool = True, +) -> Prompt[T]: ... + + +def select( + message: str, + choices: Sequence[str | Choice[T] | Separator], + *, + default: str | None = None, + use_shortcuts: bool = False, + use_arrow_keys: bool = True, + use_jk_keys: bool = True, + use_search_filter: bool = False, + show_description: bool = True, +) -> Prompt[str] | Prompt[T]: + return Prompt( + _select_impl( + message=message, + choices=[_convert_choice(choice) for choice in choices], + default=default, + use_shortcuts=use_shortcuts, + use_arrow_keys=use_arrow_keys, + use_jk_keys=use_jk_keys, + use_search_filter=use_search_filter, + show_description=show_description, + ), + message=message, + ) + + +@overload +def checkbox( + message: str, + choices: Sequence[str | Separator], + *, + validate: _ValidateObjects = _validate_objects_always, + use_jk_keys: bool = True, + use_search_filter: bool = False, + show_description: bool = True, +) -> Prompt[list[str]]: ... + + +@overload +def checkbox( + message: str, + choices: Sequence[Choice[T] | Separator], + *, + validate: _ValidateObjects = _validate_objects_always, + use_jk_keys: bool = True, + use_search_filter: bool = False, + show_description: bool = True, +) -> Prompt[list[T]]: ... + + +def checkbox( + message: str, + choices: Sequence[str | Choice[T] | Separator], + *, + validate: _ValidateObjects = _validate_objects_always, + use_jk_keys: bool = True, + use_search_filter: bool = False, + show_description: bool = True, +) -> Prompt[list[str]] | Prompt[list[T]]: + return Prompt( + _checkbox_impl( + message=message, + choices=[_convert_choice(choice) for choice in choices], + validate=validate, + use_jk_keys=use_jk_keys, + use_search_filter=use_search_filter, + show_description=show_description, + ), + message=message, + ) + + +def text(message: str, *, default: str = "", validate: _ValidateStr = _validate_str_always) -> Prompt[str]: + return Prompt(_text_impl(message=message, default=default, validate=validate), message=message) + + +def confirm(message: str, *, default: bool = True, auto_enter: bool = True) -> Prompt[bool]: + return Prompt(_confirm_impl(message=message, default=default, auto_enter=auto_enter), message=message) + + +def path( + message: str, + *, + default: str = "", + validate: _ValidateStr = _validate_str_always, + file_filter: Callable[[str], bool] | None = None, +) -> Prompt[str]: + return Prompt( + _path_impl(message=message, default=default, validate=validate, file_filter=file_filter), message=message + ) diff --git a/tariff_fetch/urdb/arcadia/build.py b/tariff_fetch/urdb/arcadia/build.py index 114da06..01807e2 100644 --- a/tariff_fetch/urdb/arcadia/build.py +++ b/tariff_fetch/urdb/arcadia/build.py @@ -1,11 +1,12 @@ """Build URDB-style output from Arcadia tariff data for a conversion scenario.""" import logging -from typing import cast +from collections.abc import Callable -import questionary +import typer from tariff_fetch.arcadia.api import ArcadiaSignalAPI +from tariff_fetch.questionary_typed import confirm from tariff_fetch.urdb.arcadia.library import Library from tariff_fetch.urdb.schema import URDBRate @@ -17,24 +18,25 @@ logger = logging.getLogger(__name__) -def build_urdb(api: ArcadiaSignalAPI, scenario: Scenario) -> URDBRate: +def build_urdb(api: ArcadiaSignalAPI, scenario: Scenario, *, interactive_errors: bool = True) -> URDBRate: """Build a URDB record by combining energy, fixed-charge, and metadata chunks.""" - library = Library(api) - try: - energy_schedule = build_energy_schedule(scenario, library) - except Exception as e: - energy_schedule = _confirm_proceed(e, "energy rate strucutre") - - try: - fixed_charge = build_fixed_charge(scenario, library) - except Exception as e: - fixed_charge = _confirm_proceed(e, "fixed charges") - - try: - metadata = build_metadata(scenario, library) - except Exception as e: - metadata = _confirm_proceed(e, "metadata") + library = Library(api, properties=scenario.properties) + energy_schedule = _build_chunk( + lambda: build_energy_schedule(scenario, library), + "energy rate strucutre", + interactive_errors=interactive_errors, + ) + fixed_charge = _build_chunk( + lambda: build_fixed_charge(scenario, library), + "fixed charges", + interactive_errors=interactive_errors, + ) + metadata = _build_chunk( + lambda: build_metadata(scenario, library), + "metadata", + interactive_errors=interactive_errors, + ) if hasattr(library, "iter_issues"): for issue in library.iter_issues(): @@ -43,15 +45,31 @@ def build_urdb(api: ArcadiaSignalAPI, scenario: Scenario) -> URDBRate: return {**energy_schedule, **fixed_charge, **metadata} -def _confirm_proceed(e: Exception, processing: str) -> URDBRate: +def _build_chunk( + builder: Callable[[], URDBRate], + processing: str, + *, + interactive_errors: bool, +) -> URDBRate: + """Run one converter stage with shared cancellation and recovery behavior.""" + + try: + return builder() + except typer.Exit: + raise + except Exception as e: + return _confirm_proceed(e, processing, interactive_errors=interactive_errors) + + +def _confirm_proceed(e: Exception, processing: str, *, interactive_errors: bool) -> URDBRate: """Ask whether to continue after a chunk-level conversion failure.""" - response = cast( - bool | None, - questionary.confirm(f"Error while converting: {processing}: {e}. Continue or print traceback and exit?").ask(), - ) - if response is None: - exit() + if not interactive_errors: + raise e from None + + response = confirm( + f"Error while converting: {processing}: {e}. Continue or print traceback and exit?" + ).ask_or_exit() if response: return {} raise e from None diff --git a/tariff_fetch/urdb/arcadia/library.py b/tariff_fetch/urdb/arcadia/library.py index 24db4c4..aaeb008 100644 --- a/tariff_fetch/urdb/arcadia/library.py +++ b/tariff_fetch/urdb/arcadia/library.py @@ -1,6 +1,7 @@ """Arcadia data access, caching, prompting, and debug persistence for conversion.""" import json +import re from datetime import date, datetime from pathlib import Path from typing import Literal, final, overload @@ -240,7 +241,7 @@ def __init__( self.debug_store = LibraryDebugStore(debug_root) self.tariffs = TariffLibrary(api, self.debug_store) self.variables = VariablePropertyLibrary(api, self.debug_store) - self._properies: dict[str, PropertyValue] = properties or {} + self._properies: dict[str, PropertyValue] = properties if properties is not None else {} self._issues: dict[tuple[object, ...], str] = {} def has_property(self, key: str) -> bool: @@ -287,71 +288,49 @@ def get_property(self, key: str, data_type: Literal["DEMAND"]) -> float: ... def get_property(self, key: str, data_type: TariffPropertyPrunedDataType) -> PropertyValue: """Return a property value, prompting and caching it if needed.""" - if (found := self._get_property(key, data_type)) is not None: - return found tariff_property = self.tariffs.get_property(key) + if (found := self._get_property(tariff_property, data_type)) is not None: + return found result = _prompt_property(tariff_property) if result is None: raise ConversionError("Property not set") - self._properies[key] = result + self._properies[tariff_property["key_name"]] = result self.debug_store.save_property_value(key, result) return result @overload - def _get_property(self, key: str, data_type: Literal["STRING"]) -> str | None: ... + def _get_property(self, tariff_property: TariffPropertyStandard, data_type: Literal["STRING"]) -> str | None: ... @overload - def _get_property(self, key: str, data_type: Literal["CHOICE"]) -> list[str] | None: ... + def _get_property( + self, tariff_property: TariffPropertyStandard, data_type: Literal["CHOICE"] + ) -> list[str] | None: ... @overload - def _get_property(self, key: str, data_type: Literal["BOOLEAN"]) -> bool | None: ... + def _get_property(self, tariff_property: TariffPropertyStandard, data_type: Literal["BOOLEAN"]) -> bool | None: ... @overload - def _get_property(self, key: str, data_type: Literal["DATE"]) -> date | None: ... + def _get_property(self, tariff_property: TariffPropertyStandard, data_type: Literal["DATE"]) -> date | None: ... @overload - def _get_property(self, key: str, data_type: Literal["DECIMAL"]) -> float | None: ... + def _get_property(self, tariff_property: TariffPropertyStandard, data_type: Literal["DECIMAL"]) -> float | None: ... @overload - def _get_property(self, key: str, data_type: Literal["INTEGER"]) -> int | None: ... + def _get_property(self, tariff_property: TariffPropertyStandard, data_type: Literal["INTEGER"]) -> int | None: ... @overload - def _get_property(self, key: str, data_type: Literal["DEMAND"]) -> float | None: ... + def _get_property(self, tariff_property: TariffPropertyStandard, data_type: Literal["DEMAND"]) -> float | None: ... - def _get_property(self, key: str, data_type: TariffPropertyPrunedDataType) -> PropertyValue | None: + def _get_property( + self, tariff_property: TariffPropertyStandard, data_type: TariffPropertyPrunedDataType + ) -> PropertyValue | None: """Return a cached property value after validating its expected data type.""" + key = tariff_property["key_name"] + override_key = _find_property_override_key(self._properies, tariff_property) try: - value = self._properies[key] + value = self._properies[override_key] except KeyError: return None - value_type = type(value) - match data_type: - case "STRING": - if not isinstance(value, str): - raise ConversionError(f"Value for property {key} is expected to be `str`, not {value_type}") - return value - case "CHOICE": - if not isinstance(value, list): - raise ConversionError(f"Value for property {key} is expected to be a list, not {value_type}") - return value - case "BOOLEAN": - if not isinstance(value, bool): - raise ConversionError(f"Value for property {key} is expected to be `bool`, not {value_type}") - return value - case "DATE": - if not isinstance(value, date): - raise ConversionError(f"Value for property {key} is expected to be `bool`, not {value_type}") - return value - case "DECIMAL": - if not isinstance(value, float): - raise ConversionError(f"Value for property {key} is expected to be `float`, not {value_type}") - return value - case "INTEGER": - if not isinstance(value, int): - raise ConversionError(f"Value for property {key} is expected to be `int`, not {value_type}") - return value - - case "DEMAND": - if not isinstance(value, float): - raise ConversionError(f"Value for property {key} is expected to be `float`, not {value_type}") - return value + coerced = _coerce_property_value(key, value, data_type, tariff_property) + self._properies[key] = coerced + return coerced def _is_tariff_effective_on(tariff: TariffExtended, dt: date) -> bool: @@ -362,6 +341,131 @@ def _is_tariff_effective_on(tariff: TariffExtended, dt: date) -> bool: return effective_date <= dt < end_date +def _normalize_alias(value: str) -> str: + """Normalize user-facing property aliases for forgiving CLI matching.""" + + return re.sub(r"[^a-z0-9]+", "", value.lower()) + + +def _find_property_override_key(properties: dict[str, PropertyValue], tariff_property: TariffPropertyStandard) -> str: + """Resolve a CLI-supplied property override key to the canonical Arcadia property key.""" + + key_name = tariff_property["key_name"] + display_name = tariff_property["display_name"] + if key_name in properties: + return key_name + if display_name in properties: + return display_name + + aliases = {_normalize_alias(key_name), _normalize_alias(display_name)} + matches = [candidate for candidate in properties if _normalize_alias(candidate) in aliases] + if not matches: + return key_name + if len(matches) > 1: + raise ConversionError( + f"Multiple property overrides matched {key_name}: {', '.join(sorted(matches))}. " + + "Use the canonical Arcadia property key to disambiguate." + ) + return matches[0] + + +def _coerce_choice_aliases(_key: str, values: list[str], tariff_property: TariffPropertyStandard) -> list[str]: + """Map CHOICE property display values back to their Arcadia machine values.""" + + choices = tariff_property.get("choices") or [] + if not choices: + return values + + by_value = {choice["value"]: choice["value"] for choice in choices} + by_display = {choice["display_value"]: choice["value"] for choice in choices} + by_normalized = {_normalize_alias(choice["display_value"]): choice["value"] for choice in choices} + + resolved: list[str] = [] + for value in values: + if value in by_value: + resolved.append(by_value[value]) + continue + if value in by_display: + resolved.append(by_display[value]) + continue + normalized = _normalize_alias(value) + if normalized in by_normalized: + resolved.append(by_normalized[normalized]) + continue + resolved.append(value) + return resolved + + +def _coerce_property_value( + key: str, + value: PropertyValue, + data_type: TariffPropertyPrunedDataType, + tariff_property: TariffPropertyStandard, +) -> PropertyValue: + """Convert CLI-supplied property overrides into the expected Arcadia property type.""" + + match data_type: + case "STRING": + if isinstance(value, str): + return value + raise ConversionError(f"Value for property {key} is expected to be `str`, not {type(value)}") + case "CHOICE": + if isinstance(value, list): + return _coerce_choice_aliases(key, value, tariff_property) + if isinstance(value, str): + return _coerce_choice_aliases( + key, [item.strip() for item in value.split(",") if item.strip()], tariff_property + ) + raise ConversionError(f"Value for property {key} is expected to be `list[str]`, not {type(value)}") + case "BOOLEAN": + if isinstance(value, bool): + return value + if isinstance(value, str): + normalized = value.strip().lower() + if normalized in {"true", "1", "yes", "y", "on"}: + return True + if normalized in {"false", "0", "no", "n", "off"}: + return False + raise ConversionError(f"Value for property {key} is expected to be `bool`, not {type(value)}") + case "DATE": + if isinstance(value, date): + return value + if isinstance(value, str): + try: + return date.fromisoformat(value) + except ValueError as e: + raise ConversionError(f"Value for property {key} is not a valid ISO date: {value}") from e + raise ConversionError(f"Value for property {key} is expected to be `date`, not {type(value)}") + case "DECIMAL" | "DEMAND": + if isinstance(value, bool): + raise ConversionError(f"Value for property {key} is expected to be `float`, not {type(value)}") + if isinstance(value, float): + return value + if isinstance(value, int): + return float(value) + if isinstance(value, str): + try: + return float(value) + except ValueError as e: + raise ConversionError(f"Value for property {key} is not a valid decimal: {value}") from e + raise ConversionError(f"Value for property {key} is expected to be `float`, not {type(value)}") + case "INTEGER": + if isinstance(value, bool): + raise ConversionError(f"Value for property {key} is expected to be `int`, not {type(value)}") + if isinstance(value, int): + return value + if isinstance(value, float): + if value.is_integer(): + return int(value) + raise ConversionError(f"Value for property {key} is not a whole number: {value}") + if isinstance(value, str): + try: + return int(value) + except ValueError as e: + raise ConversionError(f"Value for property {key} is not a valid integer: {value}") from e + raise ConversionError(f"Value for property {key} is expected to be `int`, not {type(value)}") + + def _prompt_property(tariff_property: TariffPropertyStandard) -> PropertyValue | None: """Prompt the user for a property value using the Arcadia property data type.""" diff --git a/tariff_fetch/urdb/arcadia/prompts.py b/tariff_fetch/urdb/arcadia/prompts.py index 17ad505..3c1780e 100644 --- a/tariff_fetch/urdb/arcadia/prompts.py +++ b/tariff_fetch/urdb/arcadia/prompts.py @@ -3,8 +3,8 @@ from datetime import date from typing import cast, get_args -import questionary - +from tariff_fetch import questionary_typed as q +from tariff_fetch._cli import console from tariff_fetch.arcadia.schema.common import RateChargeClass from tariff_fetch.arcadia.schema.tariffproperty import TariffPropertyStandard @@ -38,41 +38,35 @@ def prompt_charge_classes() -> set[RateChargeClass] | None: """Prompt for the Arcadia charge classes to include in conversion.""" choices = cast(tuple[RateChargeClass, ...], get_args(RateChargeClass)) - result_raw = cast( - list[RateChargeClass] | None, - questionary.checkbox( - "Select charge classes", - choices=[ - questionary.Choice( - title=choice, - value=choice, - checked=True, - ) - for choice in choices - ], - ).ask(), - ) - if result_raw is None: - return None - return set(result_raw) + result_raw = q.checkbox( + "Select charge classes", + choices=[ + q.Choice( + title=choice, + value=choice, + checked=True, + ) + for choice in choices + ], + ).ask_or_exit() + return cast(set[RateChargeClass], set(result_raw)) def prompt_string(tariff_property: TariffPropertyStandard) -> str | None: """Prompt for a string-valued Arcadia tariff property.""" + _print_property_prompt_context(tariff_property) default_value = tariff_property.get("property_value") if tariff_property["is_default"] else None - return cast( - str | None, - questionary.text( - _get_property_msg(tariff_property), - default=default_value or "", - ).ask(), - ) + return q.text( + _get_property_title(tariff_property), + default=default_value or "", + ).ask_or_exit() def prompt_choice(tariff_property: TariffPropertyStandard) -> list[str] | None: """Prompt for a multi-select Arcadia CHOICE property.""" + _print_property_prompt_context(tariff_property) if tariff_property["is_default"]: default_value_raw = tariff_property.get("property_value") default_value = {item.strip() for item in (default_value_raw or "").split(",")} @@ -80,31 +74,24 @@ def prompt_choice(tariff_property: TariffPropertyStandard) -> list[str] | None: default_value: set[str] = set() if not (choices := tariff_property.get("choices")): raise ValueError("Expected a list of choices for CHOICE property") - return cast( - list[str] | None, - questionary.checkbox( - _get_property_msg(tariff_property), - choices=[ - questionary.Choice( - title=item["display_value"], value=item["value"], checked=item["value"] in default_value - ) - for item in choices - ], - ).ask(), - ) + return q.checkbox( + _get_property_title(tariff_property), + choices=[ + q.Choice(title=item["display_value"], value=item["value"], checked=item["value"] in default_value) + for item in choices + ], + ).ask_or_exit() def prompt_boolean(tariff_property: TariffPropertyStandard) -> bool | None: """Prompt for a boolean Arcadia tariff property.""" + _print_property_prompt_context(tariff_property) default_value = tariff_property.get("property_value") if tariff_property["is_default"] else None default_value = True if default_value == "true" else (False if default_value == "false" else None) - result = cast( - bool | None, - questionary.confirm( - _get_property_msg(tariff_property), default=default_value if default_value is not None else False - ).ask(), - ) + result = q.confirm( + _get_property_title(tariff_property), default=default_value if default_value is not None else False + ).ask_or_exit() return result @@ -117,38 +104,30 @@ def prompt_date(tariff_property: TariffPropertyStandard) -> date | None: # pyri def prompt_decimal(tariff_property: TariffPropertyStandard) -> float | None: """Prompt for a decimal-valued Arcadia tariff property.""" + _print_property_prompt_context(tariff_property) default_value = tariff_property.get("property_value") if tariff_property["is_default"] else None default_value = float(default_value) if default_value else None - result_str = cast( - str | None, - questionary.text( - _get_property_msg(tariff_property), - default=str(default_value) if default_value else "", - validate=_is_float, - ).ask(), - ) - if result_str is None: - return None + result_str = q.text( + _get_property_title(tariff_property), + default=str(default_value) if default_value else "", + validate=_is_float, + ).ask_or_exit() return float(result_str) def prompt_integer(tariff_property: TariffPropertyStandard) -> float | None: """Prompt for an integer-valued Arcadia tariff property.""" + _print_property_prompt_context(tariff_property) default_value = tariff_property.get("property_value") if tariff_property["is_default"] else None default_value = int(default_value) if default_value else None - result_str = cast( - str | None, - questionary.text( - _get_property_msg(tariff_property), - default=str(default_value) if default_value else "", - validate=_is_int, - ).ask(), - ) - if result_str is None: - return None + result_str = q.text( + _get_property_title(tariff_property), + default=str(default_value) if default_value else "", + validate=_is_int, + ).ask_or_exit() return float(result_str) @@ -178,9 +157,14 @@ def _is_int(value: str) -> bool: return True -def _get_property_msg(tariff_property: TariffPropertyStandard) -> str: - """Build a prompt label from an Arcadia property's display name and description.""" +def _get_property_title(tariff_property: TariffPropertyStandard) -> str: + """Return the primary property label shown in the actual prompt.""" + + return f"[{tariff_property['key_name']}] {tariff_property['display_name']}" + + +def _print_property_prompt_context(tariff_property: TariffPropertyStandard) -> None: + """Print styled property metadata before prompting for its value.""" - title = tariff_property["display_name"] description = tariff_property["description"] - return f"{title} ({description})" + console.print(f"[dim]{description}[/dim]") diff --git a/tariff_fetch/urdb/arcadia/scenario.py b/tariff_fetch/urdb/arcadia/scenario.py index eaee35d..db83f3c 100644 --- a/tariff_fetch/urdb/arcadia/scenario.py +++ b/tariff_fetch/urdb/arcadia/scenario.py @@ -1,11 +1,13 @@ """Scenario inputs that control one Arcadia-to-URDB conversion run.""" from dataclasses import dataclass, field +from datetime import date from typing import cast, get_args from tariff_fetch.arcadia.schema.common import RateChargeClass _CHARGE_CLASSES = cast(tuple[RateChargeClass, ...], get_args(RateChargeClass)) +ScenarioPropertyValue = str | list[str] | bool | date | float | int @dataclass(frozen=True) @@ -16,3 +18,4 @@ class Scenario: year: int apply_percentages: bool charge_classes: set[RateChargeClass] = field(default_factory=set) + properties: dict[str, ScenarioPropertyValue] = field(default_factory=dict) diff --git a/tests/test_arcadia_api.py b/tests/test_arcadia_api.py new file mode 100644 index 0000000..ae9278e --- /dev/null +++ b/tests/test_arcadia_api.py @@ -0,0 +1,52 @@ +from types import SimpleNamespace + +import pytest +import requests + +from tariff_fetch.arcadia.api import ArcadiaSignalAPI, ArcadiaSignalAPIAuth + + +def test_request_raises_http_error_with_arcadia_message(): + response = requests.Response() + response.status_code = 403 + response.url = "https://api.genability.com/rest/public/tariffs" + response.request = requests.Request("GET", response.url).prepare() + response._content = b"""{ + "status": "error", + "count": 1, + "type": "Error", + "results": [ + { + "code": "tariffLimit", + "message": "Unique tariff (MTIDs) limit reached. Please contact sales (arcsales@arcadia.com) for more information.", + "objectName": "OrgUsage", + "propertyName": "tariffLimit" + } + ], + "pageCount": 25, + "pageStart": 0 + }""" + response.encoding = "utf-8" + + session = SimpleNamespace(get=lambda *args, **kwargs: response) + api = ArcadiaSignalAPI(auth=ArcadiaSignalAPIAuth("id", "key"), session=session) # type: ignore[arg-type] + + with pytest.raises(requests.HTTPError, match="Unique tariff \\(MTIDs\\) limit reached") as exc_info: + api._request("tariffs") + + assert exc_info.value.response is response + + +def test_request_preserves_default_http_error_when_arcadia_message_missing(): + response = requests.Response() + response.status_code = 403 + response.url = "https://api.genability.com/rest/public/tariffs" + response.request = requests.Request("GET", response.url).prepare() + response._content = b'{"status":"error","results":[{"code":"tariffLimit"}]}' + response.encoding = "utf-8" + + session = SimpleNamespace(get=lambda *args, **kwargs: response) + api = ArcadiaSignalAPI(auth=ArcadiaSignalAPIAuth("id", "key"), session=session) # type: ignore[arg-type] + + with pytest.raises(requests.HTTPError, match="403 Client Error"): + api._request("tariffs") diff --git a/tests/test_arcadia_cli.py b/tests/test_arcadia_cli.py new file mode 100644 index 0000000..a41fd1a --- /dev/null +++ b/tests/test_arcadia_cli.py @@ -0,0 +1,120 @@ +from datetime import date +from pathlib import Path +from types import SimpleNamespace + +from tariff_fetch._cli import genability +from tariff_fetch._cli.types import Utility + + +def test_process_genability_prints_replay_command(monkeypatch, tmp_path: Path): + printed: list[str] = [] + + monkeypatch.setattr(genability, "load_dotenv", lambda: None) + monkeypatch.setattr(genability, "os", SimpleNamespace(getenv=lambda key: "set")) + monkeypatch.setattr(genability, "ArcadiaSignalAPI", lambda: object()) + monkeypatch.setattr( + genability, "q", SimpleNamespace(confirm=lambda message: SimpleNamespace(ask_or_exit=lambda: True)) + ) + monkeypatch.setattr(genability, "_find_utility_lse_id", lambda api, utility: 1) + monkeypatch.setattr(genability, "_select_customer_classes", lambda: ["RESIDENTIAL"]) + monkeypatch.setattr(genability, "_select_tariff_types", lambda: ["DEFAULT"]) + monkeypatch.setattr( + genability, + "_select_tariffs", + lambda api, lse_id, customer_classes, tariff_types, effective_on: [("Tariff", 123)], + ) + monkeypatch.setattr( + genability, + "_fetch_tariffs", + lambda api, tariffs, effective_on: [{"master_tariff_id": 123, "tariff_name": "Tariff"}], + ) + monkeypatch.setattr( + genability, "prompt_filename", lambda output_folder, suggested_filename, ext: tmp_path / "out.json" + ) + monkeypatch.setattr(genability.console, "print", lambda message, *args, **kwargs: printed.append(str(message))) + + genability.process_genability( + utility=Utility(eia_id=1, name="Utility"), + output_folder=tmp_path, + effective_on=date(2025, 6, 1), + ) + + replay_lines = [line for line in printed if line.startswith("tariff-fetch ni arcadia ")] + assert replay_lines == ["tariff-fetch ni arcadia 123 2025-06-01"] + + +def test_process_genability_prints_multiple_replay_commands(monkeypatch, tmp_path: Path): + printed: list[str] = [] + + monkeypatch.setattr(genability, "load_dotenv", lambda: None) + monkeypatch.setattr(genability, "os", SimpleNamespace(getenv=lambda key: "set")) + monkeypatch.setattr(genability, "ArcadiaSignalAPI", lambda: object()) + monkeypatch.setattr( + genability, "q", SimpleNamespace(confirm=lambda message: SimpleNamespace(ask_or_exit=lambda: True)) + ) + monkeypatch.setattr(genability, "_find_utility_lse_id", lambda api, utility: 1) + monkeypatch.setattr(genability, "_select_customer_classes", lambda: ["RESIDENTIAL"]) + monkeypatch.setattr(genability, "_select_tariff_types", lambda: ["DEFAULT"]) + monkeypatch.setattr( + genability, + "_select_tariffs", + lambda api, lse_id, customer_classes, tariff_types, effective_on: [("Tariff A", 123), ("Tariff B", 456)], + ) + monkeypatch.setattr( + genability, + "_fetch_tariffs", + lambda api, tariffs, effective_on: [ + {"master_tariff_id": 123, "tariff_name": "Tariff A"}, + {"master_tariff_id": 456, "tariff_name": "Tariff B"}, + ], + ) + monkeypatch.setattr( + genability, "prompt_filename", lambda output_folder, suggested_filename, ext: tmp_path / "out.json" + ) + monkeypatch.setattr(genability.console, "print", lambda message, *args, **kwargs: printed.append(str(message))) + + genability.process_genability( + utility=Utility(eia_id=1, name="Utility"), + output_folder=tmp_path, + effective_on=date(2025, 6, 1), + ) + + replay_lines = [line for line in printed if line.startswith("tariff-fetch ni arcadia ")] + assert replay_lines == ["tariff-fetch ni arcadia 123 2025-06-01", "tariff-fetch ni arcadia 456 2025-06-01"] + + +def test_process_genability_stops_when_replay_proceed_is_declined(monkeypatch, tmp_path: Path): + printed: list[str] = [] + fetched: dict[str, object] = {} + + monkeypatch.setattr(genability, "load_dotenv", lambda: None) + monkeypatch.setattr(genability, "os", SimpleNamespace(getenv=lambda key: "set")) + monkeypatch.setattr(genability, "ArcadiaSignalAPI", lambda: object()) + monkeypatch.setattr( + genability, "q", SimpleNamespace(confirm=lambda message: SimpleNamespace(ask_or_exit=lambda: False)) + ) + monkeypatch.setattr(genability, "_find_utility_lse_id", lambda api, utility: 1) + monkeypatch.setattr(genability, "_select_customer_classes", lambda: ["RESIDENTIAL"]) + monkeypatch.setattr(genability, "_select_tariff_types", lambda: ["DEFAULT"]) + monkeypatch.setattr( + genability, + "_select_tariffs", + lambda api, lse_id, customer_classes, tariff_types, effective_on: [("Tariff", 123)], + ) + monkeypatch.setattr( + genability, + "_fetch_tariffs", + lambda api, tariffs, effective_on: fetched.update({"called": True}), + ) + monkeypatch.setattr( + genability, "prompt_filename", lambda output_folder, suggested_filename, ext: tmp_path / "out.json" + ) + monkeypatch.setattr(genability.console, "print", lambda message, *args, **kwargs: printed.append(str(message))) + + genability.process_genability( + utility=Utility(eia_id=1, name="Utility"), + output_folder=tmp_path, + effective_on=date(2025, 6, 1), + ) + + assert fetched == {} diff --git a/tests/test_arcadia_urdb_boundaries_build.py b/tests/test_arcadia_urdb_boundaries_build.py index dd2bc3f..e598fc8 100644 --- a/tests/test_arcadia_urdb_boundaries_build.py +++ b/tests/test_arcadia_urdb_boundaries_build.py @@ -307,7 +307,7 @@ def test_tariff_iter_rates_for_dt_records_ignored_calendar_issues_once(): def test_build_urdb_merges_converter_chunks(monkeypatch): scenario = Scenario(123, 2025, apply_percentages=True, charge_classes={"SUPPLY"}) - monkeypatch.setattr(build_mod, "Library", lambda api: SimpleNamespace()) + monkeypatch.setattr(build_mod, "Library", lambda api, properties=None: SimpleNamespace()) monkeypatch.setattr( build_mod, "build_energy_schedule", diff --git a/tests/test_arcadia_urdb_cli.py b/tests/test_arcadia_urdb_cli.py new file mode 100644 index 0000000..383f8d1 --- /dev/null +++ b/tests/test_arcadia_urdb_cli.py @@ -0,0 +1,109 @@ +from pathlib import Path +from types import SimpleNamespace + +from tariff_fetch._cli import arcadia_urdb +from tariff_fetch._cli.types import Utility + + +def test_process_genability_prints_replay_command(monkeypatch, tmp_path: Path): + printed: list[str] = [] + + monkeypatch.setattr(arcadia_urdb, "load_dotenv", lambda: None) + monkeypatch.setattr(arcadia_urdb, "ArcadiaSignalAPI", lambda: object()) + monkeypatch.setattr(arcadia_urdb, "_find_utility_lse_id", lambda api, utility: 1) + monkeypatch.setattr(arcadia_urdb, "_select_customer_classes", lambda: ["RESIDENTIAL"]) + monkeypatch.setattr(arcadia_urdb, "_select_tariff_types", lambda: ["DEFAULT"]) + monkeypatch.setattr( + arcadia_urdb, "_select_tariffs", lambda api, lse_id, customer_classes, tariff_types, year: [("Tariff", 123)] + ) + monkeypatch.setattr( + arcadia_urdb, + "_fetch_tariffs", + lambda api, tariffs, year: [ + { + "master_tariff_id": 123, + "tariff_name": "Tariff", + "properties": [ + {"key_name": "territoryId", "display_name": "Territory"}, + {"key_name": "netMetering", "display_name": "Net Metering"}, + ], + } + ], + ) + monkeypatch.setattr( + arcadia_urdb, + "q", + SimpleNamespace(confirm=lambda message: SimpleNamespace(ask_or_exit=lambda: False)), + ) + monkeypatch.setattr(arcadia_urdb, "prompt_charge_classes", lambda: {"SUPPLY", "TAX"}) + monkeypatch.setattr(arcadia_urdb, "_prompt_tariff_name", lambda default: default) + monkeypatch.setattr( + arcadia_urdb, "prompt_filename", lambda output_folder, suggested_filename, ext: tmp_path / "out.json" + ) + monkeypatch.setattr(arcadia_urdb.console, "print", lambda message, *args, **kwargs: printed.append(str(message))) + + def fake_build_urdb(api, scenario, *, interactive_errors): + scenario.properties["Territory"] = ["1", "2"] + scenario.properties["netMetering"] = True + return {"name": "Tariff"} + + monkeypatch.setattr(arcadia_urdb, "build_urdb", fake_build_urdb) + + arcadia_urdb.process_genability( + utility=Utility(eia_id=1, name="Utility"), + output_folder=tmp_path, + year=2025, + interactive_errors=True, + ) + + replay_lines = [line for line in printed if line.startswith("tariff-fetch urdb ni ")] + assert replay_lines == [ + "tariff-fetch urdb ni 123 2025 --no-apply-percentages -cc St --property netMetering=true --property territoryId=1 --property territoryId=2" + ] + + +def test_process_genability_prints_multiple_replay_commands_without_default_charge_class_flag( + monkeypatch, tmp_path: Path +): + printed: list[str] = [] + + monkeypatch.setattr(arcadia_urdb, "load_dotenv", lambda: None) + monkeypatch.setattr(arcadia_urdb, "ArcadiaSignalAPI", lambda: object()) + monkeypatch.setattr(arcadia_urdb, "_find_utility_lse_id", lambda api, utility: 1) + monkeypatch.setattr(arcadia_urdb, "_select_customer_classes", lambda: ["RESIDENTIAL"]) + monkeypatch.setattr(arcadia_urdb, "_select_tariff_types", lambda: ["DEFAULT"]) + monkeypatch.setattr( + arcadia_urdb, + "_select_tariffs", + lambda api, lse_id, customer_classes, tariff_types, year: [("Tariff A", 123), ("Tariff B", 456)], + ) + monkeypatch.setattr( + arcadia_urdb, + "_fetch_tariffs", + lambda api, tariffs, year: [ + {"master_tariff_id": 123, "tariff_name": "Tariff A", "properties": []}, + {"master_tariff_id": 456, "tariff_name": "Tariff B", "properties": []}, + ], + ) + monkeypatch.setattr( + arcadia_urdb, + "q", + SimpleNamespace(confirm=lambda message: SimpleNamespace(ask_or_exit=lambda: True)), + ) + monkeypatch.setattr(arcadia_urdb, "prompt_charge_classes", lambda: set(arcadia_urdb._ALL_CHARGE_CLASSES)) + monkeypatch.setattr(arcadia_urdb, "_prompt_tariff_name", lambda default: default) + monkeypatch.setattr( + arcadia_urdb, "prompt_filename", lambda output_folder, suggested_filename, ext: tmp_path / "out.json" + ) + monkeypatch.setattr(arcadia_urdb.console, "print", lambda message, *args, **kwargs: printed.append(str(message))) + monkeypatch.setattr(arcadia_urdb, "build_urdb", lambda api, scenario, *, interactive_errors: {"name": "Tariff"}) + + arcadia_urdb.process_genability( + utility=Utility(eia_id=1, name="Utility"), + output_folder=tmp_path, + year=2025, + interactive_errors=True, + ) + + replay_lines = [line for line in printed if line.startswith("tariff-fetch urdb ni ")] + assert replay_lines == ["tariff-fetch urdb ni 123 2025", "tariff-fetch urdb ni 456 2025"] diff --git a/tests/test_arcadia_urdb_library.py b/tests/test_arcadia_urdb_library.py new file mode 100644 index 0000000..e2a9e19 --- /dev/null +++ b/tests/test_arcadia_urdb_library.py @@ -0,0 +1,66 @@ +from datetime import date +from types import SimpleNamespace +from typing import Any, cast + +from tariff_fetch.urdb.arcadia.library import Library + + +def test_library_coerces_cli_property_overrides_to_expected_types(): + library = Library( + api=object(), # type: ignore[arg-type] + properties={ + "territoryId": "1,2", + "netMetering": "true", + "serviceStart": "2025-01-02", + "baselineKwh": "10.5", + "occupants": "3", + }, + ) + cast(Any, library).tariffs = SimpleNamespace( + get_property=lambda key: { + "territoryId": {"key_name": "territoryId", "display_name": "Territory", "data_type": "CHOICE"}, + "netMetering": {"key_name": "netMetering", "display_name": "Net Metering", "data_type": "BOOLEAN"}, + "serviceStart": {"key_name": "serviceStart", "display_name": "Service Start", "data_type": "DATE"}, + "baselineKwh": {"key_name": "baselineKwh", "display_name": "Baseline kWh", "data_type": "DECIMAL"}, + "occupants": {"key_name": "occupants", "display_name": "Occupants", "data_type": "INTEGER"}, + }[key] + ) + + assert library.get_property("territoryId", "CHOICE") == ["1", "2"] + assert library.get_property("netMetering", "BOOLEAN") is True + assert library.get_property("serviceStart", "DATE") == date(2025, 1, 2) + assert library.get_property("baselineKwh", "DECIMAL") == 10.5 + assert library.get_property("occupants", "INTEGER") == 3 + + +def test_library_accepts_property_and_choice_display_aliases(): + library = Library( + api=object(), # type: ignore[arg-type] + properties={"Territory": ["Primary Territory", "2"]}, + ) + cast(Any, library).tariffs = SimpleNamespace( + get_property=lambda key: { + "territoryId": { + "key_name": "territoryId", + "display_name": "Territory", + "data_type": "CHOICE", + "choices": [ + {"value": "1", "display_value": "Primary Territory", "data_value": "1"}, + {"value": "2", "display_value": "Secondary Territory", "data_value": "2"}, + ], + } + }[key] + ) + + assert library.get_property("territoryId", "CHOICE") == ["1", "2"] + + +def test_library_preserves_passed_empty_property_dict_for_prompted_values(): + properties: dict[str, object] = {} + + library = Library( + api=object(), # type: ignore[arg-type] + properties=properties, + ) + + assert properties is library._properies diff --git a/tests/test_cli.py b/tests/test_cli.py new file mode 100644 index 0000000..591f961 --- /dev/null +++ b/tests/test_cli.py @@ -0,0 +1,776 @@ +import json +from datetime import date +from pathlib import Path +from typing import cast + +from typer.testing import CliRunner + +from tariff_fetch import cli +from tariff_fetch._cli import console +from tariff_fetch._cli.types import Provider, Utility +from tariff_fetch.urdb.arcadia.scenario import Scenario + +runner = CliRunner() + + +def test_raw_command_runs_end_to_end(monkeypatch, tmp_path: Path): + utility = Utility(eia_id=101, name="Test Utility") + captured: dict[str, object] = {} + + monkeypatch.setattr(cli, "prompt_utility", lambda state: utility) + monkeypatch.setattr(console, "print", lambda *args, **kwargs: None) + + def fake_process_genability(*, utility: Utility, output_folder: Path, effective_on: date | None = None): + captured["utility"] = utility + captured["output_folder"] = output_folder + captured["effective_on"] = effective_on + + monkeypatch.setattr(cli, "process_genability", fake_process_genability) + + result = runner.invoke( + cli.app, + ["raw", "--state", "ny", "--provider", "genability", "--output-folder", str(tmp_path)], + ) + + assert result.exit_code == 0, result.stdout + assert captured == { + "utility": utility, + "output_folder": tmp_path, + "effective_on": None, + } + + +def test_raw_command_uses_prompted_provider_when_flag_is_missing(monkeypatch, tmp_path: Path): + utility = Utility(eia_id=111, name="Prompted Utility") + captured: dict[str, object] = {} + + monkeypatch.setattr(cli, "prompt_provider", lambda: Provider.OPENEI) + monkeypatch.setattr(cli, "prompt_utility", lambda state: utility) + monkeypatch.setattr(console, "print", lambda *args, **kwargs: None) + + def fake_process_openei(selected_utility: Utility, output_folder: Path, effective_on: date | None = None): + captured["utility"] = selected_utility + captured["output_folder"] = output_folder + captured["effective_on"] = effective_on + + monkeypatch.setattr(cli, "process_openei", fake_process_openei) + + result = runner.invoke( + cli.app, + ["raw", "--state", "ny", "--output-folder", str(tmp_path)], + ) + + assert result.exit_code == 0, result.stdout + assert captured == { + "utility": utility, + "output_folder": tmp_path, + "effective_on": None, + } + + +def test_raw_command_no_input_fails_before_prompt(tmp_path: Path): + result = runner.invoke( + cli.app, + ["raw", "--no-input", "--output-folder", str(tmp_path)], + ) + + assert result.exit_code == 1 + assert "Prompt requires interactive input but --no-input was set: Select state" in result.output + + +def test_default_command_passes_effective_date_to_provider(monkeypatch, tmp_path: Path): + utility = Utility(eia_id=303, name="Default Utility") + captured: dict[str, object] = {} + + monkeypatch.setattr(cli, "prompt_provider", lambda: Provider.GENABILITY) + monkeypatch.setattr(cli, "prompt_utility", lambda state: utility) + monkeypatch.setattr(console, "print", lambda *args, **kwargs: None) + + def fake_process_genability(*, utility: Utility, output_folder: Path, effective_on: date | None = None): + captured["utility"] = utility + captured["output_folder"] = output_folder + captured["effective_on"] = effective_on + + monkeypatch.setattr(cli, "process_genability", fake_process_genability) + + result = runner.invoke( + cli.app, + ["--state", "ny", "--output-folder", str(tmp_path), "--effective-date", "2025-06-01"], + ) + + assert result.exit_code == 0, result.stdout + assert captured == { + "utility": utility, + "output_folder": tmp_path, + "effective_on": date(2025, 6, 1), + } + + +def test_urdb_command_runs_end_to_end(monkeypatch, tmp_path: Path): + utility = Utility(eia_id=202, name="URDB Utility") + captured: dict[str, object] = {} + + monkeypatch.setattr(cli, "prompt_utility", lambda state: utility) + monkeypatch.setattr(cli, "prompt_year", lambda: 2024) + monkeypatch.setattr(console, "print", lambda *args, **kwargs: None) + + def fake_process_genability_urdb( + *, + utility: Utility, + output_folder: Path, + year: int, + interactive_errors: bool, + properties: dict[str, object] | None = None, + ): + captured["utility"] = utility + captured["output_folder"] = output_folder + captured["year"] = year + captured["interactive_errors"] = interactive_errors + captured["properties"] = properties + + monkeypatch.setattr(cli, "process_genability_urdb", fake_process_genability_urdb) + + result = runner.invoke( + cli.app, + ["urdb", "--state", "wa", "--output-folder", str(tmp_path), "--property", "territoryId=1"], + ) + + assert result.exit_code == 0, result.stdout + assert captured == { + "utility": utility, + "output_folder": tmp_path, + "year": 2024, + "interactive_errors": True, + "properties": {"territoryId": "1"}, + } + + +def test_urdb_ni_command_runs_end_to_end(monkeypatch, tmp_path: Path): + output_path = tmp_path / "out.json" + captured: dict[str, object] = {} + + monkeypatch.setattr(cli, "load_dotenv", lambda: None) + monkeypatch.setattr(cli, "ArcadiaSignalAPI", lambda: object()) + monkeypatch.setattr(console, "print", lambda *args, **kwargs: None) + + def fake_build_urdb(api, scenario: Scenario, *, interactive_errors: bool): + captured["api"] = api + captured["scenario"] = scenario + captured["interactive_errors"] = interactive_errors + return {"label": "UTIL", "utility": "Utility", "name": "Tariff", "country": "USA"} + + monkeypatch.setattr(cli, "build_urdb", fake_build_urdb) + + result = runner.invoke( + cli.app, + [ + "urdb", + "ni", + "123", + "2025", + "--output", + str(output_path), + "--fail-fast", + "--property", + "territoryId=1", + "--property", + "territoryId=2", + "--property", + "netMetering=true", + ], + ) + + assert result.exit_code == 0, result.stdout + assert captured["interactive_errors"] is False + assert cast(Scenario, captured["scenario"]).properties == {"territoryId": ["1", "2"], "netMetering": "true"} + assert output_path.exists() + assert json.loads(output_path.read_text()) == { + "label": "UTIL", + "utility": "Utility", + "name": "Tariff", + "country": "USA", + } + + +def test_urdb_ni_command_supports_charge_class_shortcuts(monkeypatch, tmp_path: Path): + output_path = tmp_path / "out.json" + captured: dict[str, object] = {} + + monkeypatch.setattr(cli, "load_dotenv", lambda: None) + monkeypatch.setattr(cli, "ArcadiaSignalAPI", lambda: object()) + monkeypatch.setattr(console, "print", lambda *args, **kwargs: None) + + def fake_build_urdb(api, scenario: Scenario, *, interactive_errors: bool): + captured["scenario"] = scenario + return {"label": "UTIL", "utility": "Utility", "name": "Tariff", "country": "USA"} + + monkeypatch.setattr(cli, "build_urdb", fake_build_urdb) + + result = runner.invoke( + cli.app, + [ + "urdb", + "ni", + "123", + "2025", + "--output", + str(output_path), + "--cc", + "Stn", + ], + ) + + assert result.exit_code == 0, result.stdout + assert cast(Scenario, captured["scenario"]).charge_classes == {"SUPPLY", "TAX", "NET_EXCESS"} + + +def test_urdb_ni_command_supports_dash_cc_alias(monkeypatch, tmp_path: Path): + output_path = tmp_path / "out.json" + captured: dict[str, object] = {} + + monkeypatch.setattr(cli, "load_dotenv", lambda: None) + monkeypatch.setattr(cli, "ArcadiaSignalAPI", lambda: object()) + monkeypatch.setattr(console, "print", lambda *args, **kwargs: None) + + def fake_build_urdb(api, scenario: Scenario, *, interactive_errors: bool): + captured["scenario"] = scenario + return {"label": "UTIL", "utility": "Utility", "name": "Tariff", "country": "USA"} + + monkeypatch.setattr(cli, "build_urdb", fake_build_urdb) + + result = runner.invoke( + cli.app, + [ + "urdb", + "ni", + "123", + "2025", + "--output", + str(output_path), + "-cc", + "St", + ], + ) + + assert result.exit_code == 0, result.stdout + assert cast(Scenario, captured["scenario"]).charge_classes == {"SUPPLY", "TAX"} + + +def test_urdb_ni_command_merges_charge_class_flags(monkeypatch, tmp_path: Path): + output_path = tmp_path / "out.json" + captured: dict[str, object] = {} + + monkeypatch.setattr(cli, "load_dotenv", lambda: None) + monkeypatch.setattr(cli, "ArcadiaSignalAPI", lambda: object()) + monkeypatch.setattr(console, "print", lambda *args, **kwargs: None) + + def fake_build_urdb(api, scenario: Scenario, *, interactive_errors: bool): + captured["scenario"] = scenario + return {"label": "UTIL", "utility": "Utility", "name": "Tariff", "country": "USA"} + + monkeypatch.setattr(cli, "build_urdb", fake_build_urdb) + + result = runner.invoke( + cli.app, + [ + "urdb", + "ni", + "123", + "2025", + "--output", + str(output_path), + "--charge-class", + "SUPPLY", + "--cc", + "Dn", + ], + ) + + assert result.exit_code == 0, result.stdout + assert cast(Scenario, captured["scenario"]).charge_classes == {"SUPPLY", "DISTRIBUTION", "NET_EXCESS"} + + +def test_urdb_ni_command_rejects_invalid_charge_class_shortcuts(monkeypatch, tmp_path: Path): + output_path = tmp_path / "out.json" + + monkeypatch.setattr(cli, "load_dotenv", lambda: None) + monkeypatch.setattr(console, "print", lambda *args, **kwargs: None) + + result = runner.invoke( + cli.app, + [ + "urdb", + "ni", + "123", + "2025", + "--output", + str(output_path), + "--cc", + "Sz", + ], + ) + + assert result.exit_code == 1 + + +def test_gas_command_runs_end_to_end(monkeypatch, tmp_path: Path): + captured: dict[str, object] = {} + + monkeypatch.setattr( + cli, + "process_rateacuity_gas", + lambda output_folder, state: captured.update( + { + "output_folder": output_folder, + "state": state, + } + ), + ) + + result = runner.invoke( + cli.app, + ["gas", "--state", "tx", "--output-folder", str(tmp_path)], + ) + + assert result.exit_code == 0, result.stdout + assert captured == { + "output_folder": tmp_path, + "state": "tx", + } + + +def test_gas_urdb_command_runs_end_to_end(monkeypatch, tmp_path: Path): + captured: dict[str, object] = {} + + monkeypatch.setattr( + cli, + "process_rateacuity_gas_urdb", + lambda output_folder, state, year: captured.update( + { + "output_folder": output_folder, + "state": state, + "year": year, + } + ), + ) + + result = runner.invoke( + cli.app, + ["gas", "urdb", "--state", "tx", "--output-folder", str(tmp_path), "--year", "2025"], + ) + + assert result.exit_code == 0, result.stdout + assert captured == { + "output_folder": tmp_path, + "state": "tx", + "year": 2025, + } + + +def test_gas_ni_command_runs_end_to_end(monkeypatch, tmp_path: Path): + output_path = tmp_path / "gas_rateacuity.json" + captured: dict[str, object] = {} + fake_results = [{"schedule": "Firm Gas Service", "sections": []}] + + monkeypatch.setattr(console, "print", lambda *args, **kwargs: None) + + def fake_fetch_rateacuity_gas_tariffs(*, state: str, utility_query: str, tariff_queries: list[str]): + captured["state"] = state + captured["utility_query"] = utility_query + captured["tariff_queries"] = tariff_queries + return "Consolidated Edison Gas", fake_results + + monkeypatch.setattr(cli, "fetch_rateacuity_gas_tariffs", fake_fetch_rateacuity_gas_tariffs) + + result = runner.invoke( + cli.app, + [ + "gas", + "ni", + "ny", + "con ed gas", + "--tariff", + "firm gas service", + "--output", + str(output_path), + ], + ) + + assert result.exit_code == 0, result.stdout + assert captured == { + "state": "ny", + "utility_query": "con ed gas", + "tariff_queries": ["firm gas service"], + } + assert json.loads(output_path.read_text()) == fake_results + + +def test_gas_urdb_ni_command_runs_end_to_end(monkeypatch, tmp_path: Path): + output_path = tmp_path / "gas_rateacuity_urdb.json" + captured: dict[str, object] = {} + fake_results = [{"label": "ceg", "utility": "Consolidated Edison Gas", "name": "Firm Gas Service"}] + + monkeypatch.setattr(console, "print", lambda *args, **kwargs: None) + + def fake_fetch_rateacuity_gas_urdb_rates( + *, + state: str, + utility_query: str, + tariff_queries: list[str], + year: int, + apply_percentages: bool, + label: str | None, + sector: str, + servicetype: str, + ): + captured["state"] = state + captured["utility_query"] = utility_query + captured["tariff_queries"] = tariff_queries + captured["year"] = year + captured["apply_percentages"] = apply_percentages + captured["label"] = label + captured["sector"] = sector + captured["servicetype"] = servicetype + return "Consolidated Edison Gas", fake_results + + monkeypatch.setattr(cli, "fetch_rateacuity_gas_urdb_rates", fake_fetch_rateacuity_gas_urdb_rates) + + result = runner.invoke( + cli.app, + [ + "gas", + "urdb", + "ni", + "ny", + "con ed gas", + "--year", + "2025", + "--tariff", + "firm gas service", + "--label", + "ceg", + "--sector", + "Commercial", + "--servicetype", + "Delivery", + "--apply-percentages", + "--output", + str(output_path), + ], + ) + + assert result.exit_code == 0, result.stdout + assert captured == { + "state": "ny", + "utility_query": "con ed gas", + "tariff_queries": ["firm gas service"], + "year": 2025, + "apply_percentages": True, + "label": "ceg", + "sector": "Commercial", + "servicetype": "Delivery", + } + assert json.loads(output_path.read_text()) == {"items": fake_results} + + +def test_ni_arcadia_command_runs_end_to_end(monkeypatch, tmp_path: Path): + output_path = tmp_path / "arcadia.json" + captured: dict[str, object] = {} + fake_results = [{"master_tariff_id": 123, "tariff_name": "Example Tariff"}] + + monkeypatch.setattr(cli, "load_dotenv", lambda: None) + monkeypatch.setattr(console, "print", lambda *args, **kwargs: None) + + class FakeTypeAdapter: + def __init__(self, _type): + pass + + def dump_json(self, value, *, indent: int): + assert indent == 2 + return json.dumps(value, indent=indent).encode() + + class FakeTariffsAPI: + def iter_pages(self, **kwargs): + captured["iter_pages_kwargs"] = kwargs + return iter(fake_results) + + class FakeArcadiaSignalAPI: + def __init__(self): + self.tariffs = FakeTariffsAPI() + + monkeypatch.setattr(cli, "ArcadiaSignalAPI", FakeArcadiaSignalAPI) + monkeypatch.setattr(cli, "TypeAdapter", FakeTypeAdapter) + + result = runner.invoke( + cli.app, + ["ni", "arcadia", "123", "2025-06-01", "--output", str(output_path)], + ) + + assert result.exit_code == 0, result.stdout + assert captured["iter_pages_kwargs"] == { + "fields": "ext", + "master_tariff_id": 123, + "effective_on": date(2025, 6, 1), + "populate_properties": True, + "populate_rates": True, + } + assert output_path.exists() + assert json.loads(output_path.read_text()) == fake_results + + +def test_show_properties_command_runs_end_to_end(monkeypatch): + captured: dict[str, object] = {} + fake_results = [{"properties": [{"key_name": "territoryId"}]}] + + monkeypatch.setattr(cli, "load_dotenv", lambda: None) + monkeypatch.setattr(console, "print", lambda *args, **kwargs: None) + + def fake_fetch_arcadia_tariffs(*, master_tariff_id: int, effective_on: date, populate_rates: bool): + captured["master_tariff_id"] = master_tariff_id + captured["effective_on"] = effective_on + captured["populate_rates"] = populate_rates + return fake_results + + def fake_print_arcadia_properties(results): + captured["results"] = results + + monkeypatch.setattr(cli, "_fetch_arcadia_tariffs", fake_fetch_arcadia_tariffs) + monkeypatch.setattr(cli, "_print_arcadia_properties", fake_print_arcadia_properties) + + result = runner.invoke( + cli.app, + ["show-properties", "123", "2025-06-01"], + ) + + assert result.exit_code == 0, result.stdout + assert captured == { + "master_tariff_id": 123, + "effective_on": date(2025, 6, 1), + "populate_rates": True, + "results": fake_results, + } + + +def test_ni_openei_command_runs_end_to_end(monkeypatch, tmp_path: Path): + output_path = tmp_path / "openei.json" + captured: dict[str, object] = {} + fake_results = [{"name": "Residential Tariff", "label": "abc"}] + + monkeypatch.setattr(cli, "load_dotenv", lambda: None) + monkeypatch.setattr(console, "print", lambda *args, **kwargs: None) + + def fake_fetch_openei_tariffs( + *, eia_id: int, sector: str, detail: str, effective_on: date, labels: list[str] | None + ): + captured["eia_id"] = eia_id + captured["sector"] = sector + captured["detail"] = detail + captured["effective_on"] = effective_on + captured["labels"] = labels + return fake_results + + monkeypatch.setattr(cli, "_fetch_openei_tariffs", fake_fetch_openei_tariffs) + + result = runner.invoke( + cli.app, + [ + "ni", + "openei", + "123", + "Residential", + "2025-06-01", + "--detail", + "minimal", + "--output", + str(output_path), + ], + ) + + assert result.exit_code == 0, result.stdout + assert captured == { + "eia_id": 123, + "sector": "Residential", + "detail": "minimal", + "effective_on": date(2025, 6, 1), + "labels": None, + } + assert json.loads(output_path.read_text()) == {"items": fake_results} + + +def test_ni_openei_command_filters_by_label(monkeypatch, tmp_path: Path): + output_path = tmp_path / "openei.json" + captured: dict[str, object] = {} + fake_results = [{"name": "Residential Tariff", "label": "abc"}] + + monkeypatch.setattr(cli, "load_dotenv", lambda: None) + monkeypatch.setattr(console, "print", lambda *args, **kwargs: None) + + def fake_fetch_openei_tariffs( + *, eia_id: int, sector: str, detail: str, effective_on: date, labels: list[str] | None + ): + captured["eia_id"] = eia_id + captured["sector"] = sector + captured["detail"] = detail + captured["effective_on"] = effective_on + captured["labels"] = labels + return fake_results + + monkeypatch.setattr(cli, "_fetch_openei_tariffs", fake_fetch_openei_tariffs) + + result = runner.invoke( + cli.app, + [ + "ni", + "openei", + "123", + "Residential", + "2025-06-01", + "--label", + "abc", + "--label", + "def", + "--output", + str(output_path), + ], + ) + + assert result.exit_code == 0, result.stdout + assert captured == { + "eia_id": 123, + "sector": "Residential", + "detail": "full", + "effective_on": date(2025, 6, 1), + "labels": ["abc", "def"], + } + assert json.loads(output_path.read_text()) == {"items": fake_results} + + +def test_ni_rateacuity_command_runs_end_to_end(monkeypatch, tmp_path: Path): + output_path = tmp_path / "rateacuity.json" + captured: dict[str, object] = {} + fake_results = [{"schedule": "Residential Service", "sections": []}] + + monkeypatch.setattr(console, "print", lambda *args, **kwargs: None) + + def fake_fetch_rateacuity_tariffs(*, state: str, utility_query: str, tariff_queries: list[str]): + captured["state"] = state + captured["utility_query"] = utility_query + captured["tariff_queries"] = tariff_queries + return "Consolidated Edison Company of New York", fake_results + + monkeypatch.setattr(cli, "fetch_rateacuity_tariffs", fake_fetch_rateacuity_tariffs) + + result = runner.invoke( + cli.app, + [ + "ni", + "rateacuity", + "fuzzy", + "ny", + "con ed", + "--tariff", + "residential service", + "--tariff", + "time of use", + "--output", + str(output_path), + ], + ) + + assert result.exit_code == 0, result.stdout + assert captured == { + "state": "ny", + "utility_query": "con ed", + "tariff_queries": ["residential service", "time of use"], + } + assert json.loads(output_path.read_text()) == fake_results + + +def test_ni_rateacuity_command_supports_eia_id_lookup(monkeypatch, tmp_path: Path): + output_path = tmp_path / "rateacuity.json" + captured: dict[str, object] = {} + fake_results = [{"schedule": "Residential Service", "sections": []}] + + monkeypatch.setattr(console, "print", lambda *args, **kwargs: None) + monkeypatch.setattr( + cli, + "_get_utility_by_eia_id", + lambda eia_id: cli._UtilityLookup(eia_id=eia_id, name="Consolidated Edison", state="NY"), + ) + + def fake_fetch_rateacuity_tariffs(*, state: str, utility_query: str, tariff_queries: list[str]): + captured["state"] = state + captured["utility_query"] = utility_query + captured["tariff_queries"] = tariff_queries + return "Consolidated Edison Company of New York", fake_results + + monkeypatch.setattr(cli, "fetch_rateacuity_tariffs", fake_fetch_rateacuity_tariffs) + + result = runner.invoke( + cli.app, + [ + "ni", + "rateacuity", + "eia-id", + "123", + "--tariff", + "residential service", + "--output", + str(output_path), + ], + ) + + assert result.exit_code == 0, result.stdout + assert captured == { + "state": "ny", + "utility_query": "Consolidated Edison", + "tariff_queries": ["residential service"], + } + assert json.loads(output_path.read_text()) == fake_results + + +def test_collect_arcadia_property_rows_merges_choices_across_tariffs(): + tariffs = [ + { + "properties": [ + { + "key_name": "chargeClass", + "display_name": "Charge class", + "data_type": "CHOICE", + "description": "Handled separately", + "choices": [ + {"value": "SUPPLY", "display_value": "Supply", "data_value": "SUPPLY"}, + ], + }, + { + "key_name": "territoryId", + "display_name": "Territory", + "data_type": "CHOICE", + "description": "Select the service territory", + }, + ] + }, + { + "properties": [ + { + "key_name": "territoryId", + "display_name": "Territory", + "data_type": "CHOICE", + "description": "Select the service territory", + "choices": [ + {"value": "1", "display_value": "Primary Territory", "data_value": "1"}, + {"value": "2", "display_value": "Secondary Territory", "data_value": "2"}, + ], + } + ] + }, + ] + + assert cli._collect_arcadia_property_rows(tariffs) == { + "territoryId": ( + "Territory", + "CHOICE", + "Select the service territory", + "Primary Territory=1, Secondary Territory=2", + ) + } diff --git a/tests/test_openei_cli.py b/tests/test_openei_cli.py new file mode 100644 index 0000000..2f5b37c --- /dev/null +++ b/tests/test_openei_cli.py @@ -0,0 +1,67 @@ +from datetime import date +from pathlib import Path +from types import SimpleNamespace + +from tariff_fetch._cli import openei +from tariff_fetch._cli.types import Utility + + +def test_process_openei_prints_replay_command(monkeypatch, tmp_path: Path): + printed: list[str] = [] + + monkeypatch.setattr(openei, "load_dotenv", lambda: None) + monkeypatch.setattr(openei, "os", SimpleNamespace(getenv=lambda key: "set")) + monkeypatch.setattr(openei, "_prompt_sector", lambda: "Residential") + monkeypatch.setattr(openei, "_prompt_detail_level", lambda: "minimal") + monkeypatch.setattr( + openei, + "_get_tariffs", + lambda eia_id, sector, detail, effective_on: [ + {"name": "Tariff A", "label": "abc"}, + {"name": "Tariff B", "label": "def"}, + ], + ) + monkeypatch.setattr(openei, "_prompt_tariffs", lambda tariffs: [tariffs[0]]) + monkeypatch.setattr(openei, "prompt_filename", lambda output_folder, suggested_filename, ext: tmp_path / "out.json") + monkeypatch.setattr(openei.console, "print", lambda message, *args, **kwargs: printed.append(str(message))) + + openei.process_openei( + utility=Utility(eia_id=123, name="Utility"), + output_folder=tmp_path, + effective_on=date(2025, 6, 1), + ) + + replay_lines = [line for line in printed if line.startswith("tariff-fetch ni openei ")] + assert replay_lines == ["tariff-fetch ni openei 123 Residential 2025-06-01 --detail minimal --label abc"] + + +def test_process_openei_prints_multiple_replay_commands(monkeypatch, tmp_path: Path): + printed: list[str] = [] + + monkeypatch.setattr(openei, "load_dotenv", lambda: None) + monkeypatch.setattr(openei, "os", SimpleNamespace(getenv=lambda key: "set")) + monkeypatch.setattr(openei, "_prompt_sector", lambda: "Residential") + monkeypatch.setattr(openei, "_prompt_detail_level", lambda: "minimal") + monkeypatch.setattr( + openei, + "_get_tariffs", + lambda eia_id, sector, detail, effective_on: [ + {"name": "Tariff A", "label": "abc"}, + {"name": "Tariff B", "label": "def"}, + ], + ) + monkeypatch.setattr(openei, "_prompt_tariffs", lambda tariffs: tariffs) + monkeypatch.setattr(openei, "prompt_filename", lambda output_folder, suggested_filename, ext: tmp_path / "out.json") + monkeypatch.setattr(openei.console, "print", lambda message, *args, **kwargs: printed.append(str(message))) + + openei.process_openei( + utility=Utility(eia_id=123, name="Utility"), + output_folder=tmp_path, + effective_on=date(2025, 6, 1), + ) + + replay_lines = [line for line in printed if line.startswith("tariff-fetch ni openei ")] + assert replay_lines == [ + "tariff-fetch ni openei 123 Residential 2025-06-01 --detail minimal --label abc", + "tariff-fetch ni openei 123 Residential 2025-06-01 --detail minimal --label def", + ] diff --git a/tests/test_rateacuity_cli.py b/tests/test_rateacuity_cli.py new file mode 100644 index 0000000..01e2803 --- /dev/null +++ b/tests/test_rateacuity_cli.py @@ -0,0 +1,27 @@ +from tariff_fetch._cli.rateacuity import match_rateacuity_choice, match_rateacuity_choices + + +def test_match_rateacuity_choice_compares_case_insensitively(): + assert ( + match_rateacuity_choice( + # Fuzzy matching should ignore case for runtime dropdown text. + query="con ed", + choices=[ + "Pacific Gas and Electric Company", + "Consolidated Edison Company of New York", + ], + category="Utility", + ) + == "Consolidated Edison Company of New York" + ) + + +def test_match_rateacuity_choices_deduplicates_repeated_matches(): + assert match_rateacuity_choices( + queries=["residential service", "RESIDENTIAL"], + choices=[ + "General Service Demand", + "Residential Service", + ], + category="Tariff", + ) == ["Residential Service"] diff --git a/tests/test_rateacuity_cli_replay.py b/tests/test_rateacuity_cli_replay.py new file mode 100644 index 0000000..f108a20 --- /dev/null +++ b/tests/test_rateacuity_cli_replay.py @@ -0,0 +1,499 @@ +from pathlib import Path +from types import SimpleNamespace + +import tenacity + +from tariff_fetch._cli import rateacuity, rateacuity_gas_urdb +from tariff_fetch._cli.types import Utility + + +class _FakeAttempt: + def __enter__(self): + return None + + def __exit__(self, exc_type, exc, tb): + return False + + +class _FakeRetrying: + def __iter__(self): + yield _FakeAttempt() + + +def test_process_rateacuity_prints_replay_command(monkeypatch, tmp_path: Path): + printed: list[str] = [] + + monkeypatch.setattr(rateacuity, "load_dotenv", lambda: None) + monkeypatch.setattr(rateacuity, "os", SimpleNamespace(getenv=lambda key: "set")) + monkeypatch.setattr( + rateacuity, + "tenacity", + SimpleNamespace( + Retrying=lambda **kwargs: _FakeRetrying(), + stop_after_attempt=tenacity.stop_after_attempt, + retry_if_exception_type=tenacity.retry_if_exception_type, + ), + ) + + class FakeScrapingState: + def __init__(self): + self._current_tariff = "" + + def login(self, username, password): + return self + + def electric(self): + return self + + def benchmark_all(self): + return self + + def select_state(self, state): + return self + + def get_utilities(self): + return ["Consolidated Edison Company of New York"] + + def select_utility(self, utility): + return self + + def get_schedules(self): + return ["Residential Service", "Time of Use"] + + def select_schedule(self, tariff): + self._current_tariff = tariff + return self + + def as_sections(self): + return [{"section": self._current_tariff}] + + def back_to_selections(self): + return self + + class FakeContext: + def __enter__(self): + return object() + + def __exit__(self, exc_type, exc, tb): + return False + + monkeypatch.setattr(rateacuity, "create_context", lambda: FakeContext()) + monkeypatch.setattr(rateacuity, "LoginState", lambda context: FakeScrapingState()) + monkeypatch.setattr( + rateacuity, + "q", + SimpleNamespace( + confirm=lambda message: SimpleNamespace(ask_or_exit=lambda: True), + checkbox=lambda **kwargs: SimpleNamespace(ask_or_exit=lambda: ["Residential Service", "Time of Use"]), + select=lambda **kwargs: SimpleNamespace(ask_or_exit=lambda: "unused"), + ), + ) + monkeypatch.setattr( + rateacuity, "prompt_filename", lambda output_folder, suggested_filename, ext: tmp_path / "out.json" + ) + monkeypatch.setattr(rateacuity.console, "print", lambda message, *args, **kwargs: printed.append(str(message))) + monkeypatch.setattr(rateacuity.console, "log", lambda *args, **kwargs: None) + + rateacuity.process_rateacuity( + output_folder=tmp_path, + state="ny", + utility=Utility(eia_id=123, name="Consolidated Edison"), + ) + + replay_lines = [line for line in printed if line.startswith("tariff-fetch ni rateacuity ")] + assert replay_lines == [ + "tariff-fetch ni rateacuity eia-id 123 --tariff 'Residential Service' --tariff 'Time of Use'" + ] + + +def test_process_rateacuity_gas_prints_replay_command(monkeypatch, tmp_path: Path): + printed: list[str] = [] + + monkeypatch.setattr(rateacuity, "load_dotenv", lambda: None) + monkeypatch.setattr( + rateacuity, + "os", + SimpleNamespace( + getenv=lambda key: "set", + ), + ) + monkeypatch.setattr( + rateacuity, + "tenacity", + SimpleNamespace( + Retrying=lambda **kwargs: _FakeRetrying(), + stop_after_attempt=tenacity.stop_after_attempt, + retry_if_exception_type=tenacity.retry_if_exception_type, + ), + ) + + class FakeScrapingState: + def __init__(self): + self._current_tariff = "" + + def login(self, username, password): + return self + + def gas(self): + return self + + def benchmark_all(self): + return self + + def select_state(self, state): + return self + + def get_utilities(self): + return ["Consolidated Edison Gas"] + + def select_utility(self, utility): + return self + + def get_schedules(self): + return ["Firm Gas Service", "Interruptible Gas Service"] + + def select_schedule(self, tariff): + self._current_tariff = tariff + return self + + def as_sections(self): + return [{"section": self._current_tariff}] + + def back_to_selections(self): + return self + + class FakeContext: + def __enter__(self): + return object() + + def __exit__(self, exc_type, exc, tb): + return False + + monkeypatch.setattr(rateacuity, "create_context", lambda: FakeContext()) + monkeypatch.setattr(rateacuity, "LoginState", lambda context: FakeScrapingState()) + monkeypatch.setattr( + rateacuity, + "q", + SimpleNamespace( + confirm=lambda message: SimpleNamespace(ask_or_exit=lambda: True), + checkbox=lambda **kwargs: SimpleNamespace( + ask_or_exit=lambda: ["Firm Gas Service", "Interruptible Gas Service"] + ), + select=lambda **kwargs: SimpleNamespace(ask_or_exit=lambda: "Consolidated Edison Gas"), + ), + ) + monkeypatch.setattr( + rateacuity, "prompt_filename", lambda output_folder, suggested_filename, ext: tmp_path / "out.json" + ) + monkeypatch.setattr(rateacuity.console, "print", lambda message, *args, **kwargs: printed.append(str(message))) + monkeypatch.setattr(rateacuity.console, "log", lambda *args, **kwargs: None) + + rateacuity.process_rateacuity_gas( + output_folder=tmp_path, + state="ny", + ) + + replay_lines = [line for line in printed if line.startswith("tariff-fetch gas ni ")] + assert replay_lines == [ + "tariff-fetch gas ni ny 'Consolidated Edison Gas' --tariff 'Firm Gas Service' --tariff 'Interruptible Gas Service'" + ] + + +def test_process_rateacuity_stops_when_replay_proceed_is_declined(monkeypatch, tmp_path: Path): + printed: list[str] = [] + + monkeypatch.setattr(rateacuity, "load_dotenv", lambda: None) + monkeypatch.setattr(rateacuity, "os", SimpleNamespace(getenv=lambda key: "set")) + monkeypatch.setattr( + rateacuity, + "tenacity", + SimpleNamespace( + Retrying=lambda **kwargs: _FakeRetrying(), + stop_after_attempt=tenacity.stop_after_attempt, + retry_if_exception_type=tenacity.retry_if_exception_type, + ), + ) + + class FakeScrapingState: + def __init__(self): + self._current_tariff = "" + self.selected = False + + def login(self, username, password): + return self + + def electric(self): + return self + + def benchmark_all(self): + return self + + def select_state(self, state): + return self + + def get_utilities(self): + return ["Consolidated Edison Company of New York"] + + def select_utility(self, utility): + return self + + def get_schedules(self): + return ["Residential Service"] + + def select_schedule(self, tariff): + raise AssertionError("should not fetch tariff after declining proceed") + + def as_sections(self): + return [] + + def back_to_selections(self): + return self + + class FakeContext: + def __enter__(self): + return object() + + def __exit__(self, exc_type, exc, tb): + return False + + monkeypatch.setattr(rateacuity, "create_context", lambda: FakeContext()) + monkeypatch.setattr(rateacuity, "LoginState", lambda context: FakeScrapingState()) + monkeypatch.setattr( + rateacuity, + "q", + SimpleNamespace( + confirm=lambda message: SimpleNamespace(ask_or_exit=lambda: message != "Proceed?"), + checkbox=lambda **kwargs: SimpleNamespace(ask_or_exit=lambda: ["Residential Service"]), + select=lambda **kwargs: SimpleNamespace(ask_or_exit=lambda: "unused"), + ), + ) + monkeypatch.setattr( + rateacuity, "prompt_filename", lambda output_folder, suggested_filename, ext: tmp_path / "out.json" + ) + monkeypatch.setattr(rateacuity.console, "print", lambda message, *args, **kwargs: printed.append(str(message))) + monkeypatch.setattr(rateacuity.console, "log", lambda *args, **kwargs: None) + + rateacuity.process_rateacuity( + output_folder=tmp_path, + state="ny", + utility=Utility(eia_id=123, name="Consolidated Edison"), + ) + + +def test_process_rateacuity_gas_stops_when_replay_proceed_is_declined(monkeypatch, tmp_path: Path): + printed: list[str] = [] + + monkeypatch.setattr(rateacuity, "load_dotenv", lambda: None) + monkeypatch.setattr(rateacuity, "os", SimpleNamespace(getenv=lambda key: "set")) + monkeypatch.setattr( + rateacuity, + "tenacity", + SimpleNamespace( + Retrying=lambda **kwargs: _FakeRetrying(), + stop_after_attempt=tenacity.stop_after_attempt, + retry_if_exception_type=tenacity.retry_if_exception_type, + ), + ) + + class FakeScrapingState: + def __init__(self): + self._current_tariff = "" + + def login(self, username, password): + return self + + def gas(self): + return self + + def benchmark_all(self): + return self + + def select_state(self, state): + return self + + def get_utilities(self): + return ["Consolidated Edison Gas"] + + def select_utility(self, utility): + return self + + def get_schedules(self): + return ["Firm Gas Service"] + + def select_schedule(self, tariff): + raise AssertionError("should not fetch tariff after declining proceed") + + def as_sections(self): + return [] + + def back_to_selections(self): + return self + + class FakeContext: + def __enter__(self): + return object() + + def __exit__(self, exc_type, exc, tb): + return False + + monkeypatch.setattr(rateacuity, "create_context", lambda: FakeContext()) + monkeypatch.setattr(rateacuity, "LoginState", lambda context: FakeScrapingState()) + monkeypatch.setattr( + rateacuity, + "q", + SimpleNamespace( + confirm=lambda message: SimpleNamespace(ask_or_exit=lambda: False), + checkbox=lambda **kwargs: SimpleNamespace(ask_or_exit=lambda: ["Firm Gas Service"]), + select=lambda **kwargs: SimpleNamespace(ask_or_exit=lambda: "Consolidated Edison Gas"), + ), + ) + monkeypatch.setattr( + rateacuity, "prompt_filename", lambda output_folder, suggested_filename, ext: tmp_path / "out.json" + ) + monkeypatch.setattr(rateacuity.console, "print", lambda message, *args, **kwargs: printed.append(str(message))) + monkeypatch.setattr(rateacuity.console, "log", lambda *args, **kwargs: None) + + rateacuity.process_rateacuity_gas( + output_folder=tmp_path, + state="ny", + ) + + +def test_process_rateacuity_gas_urdb_prints_replay_commands(monkeypatch, tmp_path: Path): + printed: list[str] = [] + + monkeypatch.setattr(rateacuity_gas_urdb, "load_dotenv", lambda: None) + monkeypatch.setattr( + rateacuity_gas_urdb, + "os", + SimpleNamespace(getenv=lambda key: "set"), + ) + monkeypatch.setattr( + rateacuity_gas_urdb, + "tenacity", + SimpleNamespace( + Retrying=lambda **kwargs: _FakeRetrying(), + stop_after_attempt=tenacity.stop_after_attempt, + retry_if_exception_type=tenacity.retry_if_exception_type, + ), + ) + + class FakeHistoryData: + def __init__(self, df): + self.df = df + + def validate_rows(self): + return [] + + def get_unknown_nonempty_columns(self): + return [] + + def rows(self): + return [] + + class FakeScrapingState: + def __init__(self): + self._current_tariff = "" + + def login(self, username, password): + return self + + def gas(self): + return self + + def history(self): + return self + + def select_state(self, state): + return self + + def get_utilities(self): + return ["Consolidated Edison Gas"] + + def select_utility(self, utility): + return self + + def get_schedules(self): + return ["Firm Gas Service", "Interruptible Gas Service"] + + def select_schedule(self, tariff): + self._current_tariff = tariff + return self + + def set_enddate(self, dt): + return self + + def set_number_of_comparisons(self, n): + return self + + def set_frequency(self, n): + return self + + def as_dataframe(self): + return object() + + def back_to_selections(self): + return self + + class FakeContext: + def __enter__(self): + return object() + + def __exit__(self, exc_type, exc, tb): + return False + + label_answers = iter(["ceg", "custom"]) + sector_answers = iter(["Commercial", "Residential"]) + servicetype_answers = iter(["Delivery", "Bundled"]) + percentage_answers = iter([True, False]) + + monkeypatch.setattr(rateacuity_gas_urdb, "create_context", lambda: FakeContext()) + monkeypatch.setattr(rateacuity_gas_urdb, "LoginState", lambda context: FakeScrapingState()) + monkeypatch.setattr(rateacuity_gas_urdb, "HistoryData", FakeHistoryData) + monkeypatch.setattr(rateacuity_gas_urdb, "_get_percentage_columns", lambda rows: [("Pct", None, 1.0)]) + monkeypatch.setattr(rateacuity_gas_urdb, "build_urdb", lambda rows, apply_percentages: {}) + monkeypatch.setattr( + rateacuity_gas_urdb, + "Confirm", + SimpleNamespace(ask=lambda *args, **kwargs: next(percentage_answers)), + ) + monkeypatch.setattr( + rateacuity_gas_urdb, + "q", + SimpleNamespace( + checkbox=lambda **kwargs: SimpleNamespace( + ask_or_exit=lambda: ["Firm Gas Service", "Interruptible Gas Service"] + ), + select=lambda *args, **kwargs: SimpleNamespace( + ask_or_exit=lambda: ( + "Consolidated Edison Gas" + if kwargs.get("message") == "Select a utility from available choices" + else next( + sector_answers + if args and args[0] == "Sector" and kwargs.get("default") == "Residential" + else servicetype_answers + ) + ) + ), + text=lambda *args, **kwargs: SimpleNamespace(ask_or_exit=lambda: next(label_answers)), + ), + ) + monkeypatch.setattr( + rateacuity_gas_urdb, "prompt_filename", lambda output_folder, suggested_filename, ext: tmp_path / "out.json" + ) + monkeypatch.setattr( + rateacuity_gas_urdb.console, "print", lambda message, *args, **kwargs: printed.append(str(message)) + ) + monkeypatch.setattr(rateacuity_gas_urdb.console, "log", lambda *args, **kwargs: None) + + rateacuity_gas_urdb.process_rateacuity_gas_urdb( + output_folder=tmp_path, + state="ny", + year=2025, + ) + + replay_lines = [line for line in printed if line.startswith("tariff-fetch gas urdb ni ")] + assert replay_lines == [ + "tariff-fetch gas urdb ni ny 'Consolidated Edison Gas' --year 2025 --tariff 'Firm Gas Service' --apply-percentages --sector Commercial --servicetype Delivery", + "tariff-fetch gas urdb ni ny 'Consolidated Edison Gas' --year 2025 --tariff 'Interruptible Gas Service' --label custom", + ] diff --git a/uv.lock b/uv.lock index 37e3f24..ab3ccf1 100644 --- a/uv.lock +++ b/uv.lock @@ -1486,6 +1486,7 @@ dependencies = [ { name = "fastexcel" }, { name = "fuzzywuzzy" }, { name = "pathvalidate" }, + { name = "platformdirs" }, { name = "polars" }, { name = "pydantic" }, { name = "python-dotenv" }, @@ -1518,6 +1519,7 @@ requires-dist = [ { name = "fastexcel", specifier = ">=0.16.0" }, { name = "fuzzywuzzy", specifier = ">=0.18.0" }, { name = "pathvalidate", specifier = ">=3.3.1" }, + { name = "platformdirs", specifier = ">=4.5.0" }, { name = "polars", specifier = ">=1.34.0" }, { name = "pydantic", specifier = ">=2.12.5" }, { name = "python-dotenv", specifier = ">=1.1.1" },