add validator report

[lkstrp]

Adds Validator CI. See the report below.

• It is triggered by any new commit in pr.

• Runs on a self-hosted github runner and solves on z1, so basically any configuration can be set. No restrictions. It just takes a while to update the comment if you choose a computationally intensive config.

• Compares the base branch (main) with the latest commit in the pr branch. So runs both, but also caches (in addition to snakemake caching) based on commit hashes. - Edits one main report comment when new commits are pushed to the repo, to avoid spamming.

• Set all settings in workflow file (mainly which plots to show, pre-commands, environment selection, etc.)

• Some things which are missing:

  • Better comparison (use n.statistics)
  • Doesn't run on PRs from forks
  • Handle environment updates
  • More file filters to avoid unnecessary runs
  • Concurrent runs

Also have a look at the Wishlist and see action repo to create any issues. This will be moved to the PyPSA org at some point.

[fneum]

Let's try it out!

[github-actions]

Validator Report

I am the Validator. Download all artifacts here.
I'll be back and edit this comment for each new commit.

General

Plots comparison

Main branch	Feature branch

Files comparison

	Status	NRMSE	MAE (norm)
weighted_prices.csv	⚠️Changed	2.266	0.34
nodal_capacities.csv	⚠️ NaN mismatch
nodal_supply_energy.csv	⚠️ NaN mismatch
nodal_costs.csv	⚠️ NaN mismatch
nodal_cfs.csv	⚠️ NaN mismatch
cfs.csv	✅ Almost equal	0.039	0.09
curtailment.csv	✅ Almost equal	0.000	0.18
price_statistics.csv	✅ Almost equal	0.001	0.17
capacities.csv	✅ Almost equal	0.000	0.03
market_values.csv	✅ Almost equal	0.003	0.04
costs.csv	✅ Almost equal	0.000	0.01
metrics.csv	✅ Almost equal	0.000	0.11
prices.csv	✅ Almost equal	0.000	0.11
supply_energy.csv	✅ Almost equal	0.000	0.03
supply.csv	✅ Almost equal	0.000	0.03
energy.csv	✅ Almost equal	0.000	0.03

NRMSE: Normalized (min-max) Root Mean Square Error
MAE (norm): Mean Absolute Error on normalized Data (min-max
Status Threshold: MAE (norm) > 0.05 and NRMSE > 2

Model Metrics

Benchmarks

Comparing validator (bced672) with master (dbe7f48).
Branch is 4 commits ahead and 0 commits behind.
Last updated on 2024-09-13 15:54:13 CEST.