Investigate ensemble evaluation methods

[mgdenno]

What is needed?
Do we need to add "trace" or "member" to the timeseries table.

[samlamont]

Adding some notes on existing verification systems/packages for reference:

• scoringrules: https://github.com/frazane/scoringrules
  • Python, uses numba implementations from properscoring
  • Includes more metrics than properscoring
  • Seems like a good fit for implementation in TEEHR
• properscoring: https://github.com/properscoring/properscoring
  • CRPS and Brier Score only
  • Python, numba implementations with guvectorize
• METplus: https://dtcenter.org/community-code/model-evaluation-tools-met/documentation
  • Through NCAR's Developmental Testbed Center
  • METplus verification system
  • Seems like a framework consisting of statistical engines, visualization, data management
  • Focuses on gridded NWP models (originally focused on WRF)
  • ~25 ensemble verification stats (ENCT - Ensemble Continuous Statistics)
  • Includes python wrappers
• evalhyd: https://hydrogr.github.io/evalhyd/
  • Deterministic and Probabilistic metrics
  • Bootstrapping
  • Includes other functionalities like memoization, missing data handling, masking, transformation, diagnostics)
  • C++ core with python and R bindings
• hydrostats: https://github.com/BYU-Hydroinformatics/Hydrostats
  • Developed by BYU
  • Mostly a collection of lots of metrics, with some added functionality (viz, preprocessing)
  • Includes sample data
  • Goodness-of-fit metrics from HydroErr module: https://hydroerr.readthedocs.io/en/stable/list_of_metrics.html
  • Includes ensemble metrics
  • I found ens_crps to be ~1e5 times slower than scoringrules implementation (same results)
• hydrotools.metrics: https://noaa-owp.github.io/hydrotools/hydrotools.metrics.html
  • Does not include ensemble metrics
• NWM Ensemble Verification System: https://www.weather.gov/media/owp/oh/rfcdev/docs/EVS_MANUAL_V1.0.pdf
  • Doesn't look like the code is publicly available (java?)
  • GUI-based
• scores: https://scores.readthedocs.io/en/stable/included.html
  • Includes many continuous, probabilistic, and categorical metrics
  • Built around dask and xarray
  • Handles gridded and point-based input data