New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge in code to run and score retrospective evaluation #102

Merged

dylanhmorris merged 153 commits into main from dhm-run-scoringutils

Nov 8, 2024

Contributor

dylanhmorris commented Nov 4, 2024

Adds functionality for fetching latest evaluation data.
Adds functionality for plotting forecasts against eventually observed (evaluation) data.
Adds functionality for computing the CDC baseline baseline forecast using epipredict and scoring relative to it.
Scores individual forecasts against that evaluation data and collates score tables.
Adds functionality for producing summary plots of scores

dylanhmorris and others added 30 commits

October 31, 2024 20:51


          Postprocess now saves posterior samples as parquet

fa4065e


          Merge branch 'main' into dhm-run-scoringutils

6be6f4e


          Add scoringutils 2.0 as dep

4a63342


          Go back to using latest PMF for now while debug

eadc55e


          Checkpoint forecast scoring


          Working score for single forecast

b05cf30


          Rename loop shell scripts, add optional scoring to forecast_state.py

93fb9e4


          extra deps

559b4b5


          Fix incomplete change in variable names

44cfea9


          Add baseline to score forecast

2c5ada8


          Add missing join column

012c571


          add cdc flat forecast

6b3f264


          Add model to forecast unit

df6fd5c


          Fix bug

1ef8b76


          Add quantile scoring

495c91c


          Merge CDC baseline

9582f08


          Fix accidental deletion

afe7fe6


          Add remotes for cmu packages

185ae7d


          Preinstall CMU packages

92606f4


          Add CDC baseline to scoring

b8e6b4b


          Cache installation of pyrenew_hew itself in container

0de7af4


          Revert containerfile for now


          add cdc baseline proportion forecast, other cleanup

55cc99f


          Path fixes in scoring

7808e75


          Working scoring

75e8b18


          Style cleanup

930a16e


          Add eval data saving script

863be5e


          Add in DB's eval data pull

6c0eed5


          Only pull up to end of forecast horizon for efficiency

00aeadd


          Fix missing arg

c36e08f

damonbayer reviewed

View reviewed changes

nssp_demo/batch/setup_prod_job.py Outdated Show resolved Hide resolved

damonbayer reviewed

View reviewed changes

nssp_demo/batch/setup_prod_job.py Show resolved Hide resolved

damonbayer reviewed

View reviewed changes

nssp_demo/batch/setup_prod_job.py Outdated Show resolved Hide resolved

damonbayer reviewed

View reviewed changes

nssp_demo/collate_score_tables.R Show resolved Hide resolved

dylanhmorris and others added 2 commits

November 8, 2024 16:28


          Update nssp_demo/batch/setup_eval_job.py

30b0313

Co-authored-by: Damon Bayer <[email protected]>


          Update nssp_demo/batch/setup_prod_job.py

39394c8

Co-authored-by: Damon Bayer <[email protected]>

damonbayer reviewed

View reviewed changes

nssp_demo/collate_score_tables.R Show resolved Hide resolved


          Use itertools product for setup_prod_job.py

ee1fd98

damonbayer reviewed

View reviewed changes

nssp_demo/create_hubverse_table.R Outdated Show resolved Hide resolved


          Add explanatory comment

d9f0819

This was referenced Nov 8, 2024

Move addition of metadata to score tables to score_forecast.R #112

Open

Make get_all_forecast_dirs function part of hewr #113

Open

Discuss DRY for azure setup scripts #114

Closed


          Demystify magic number

5ce3003

damonbayer reviewed

View reviewed changes

nssp_demo/collate_score_tables.R Outdated Show resolved Hide resolved

damonbayer reviewed

View reviewed changes

nssp_demo/make_observed_data_table.py Outdated Show resolved Hide resolved

damonbayer self-requested a review

November 8, 2024 22:10

damonbayer approved these changes

View reviewed changes

Collaborator

damonbayer left a comment

Nice work @dylanhmorris! You're welcome to merge after addressing my final comments.

dylanhmorris and others added 7 commits

November 8, 2024 17:33


          Update nssp_demo/collate_score_tables.R

dd077a2

Co-authored-by: Damon Bayer <[email protected]>


          one more saveRDS-->write_rds

44519e1


          Use tribble

26b7c0d


          Fix variable name


          Fix variable name clash

0c29f2f


          Transform y, not x

c3f1407


          Try a different approach to y transforms

b28ffda

dylanhmorris merged commit b905057 into main

1 check passed

dylanhmorris deleted the dhm-run-scoringutils branch

November 8, 2024 23:35

This was referenced Nov 11, 2024

Quantile-based forecast scoring #91

Closed

A model to generate $N(t)$. #53

Closed

damonbayer mentioned this pull request

Separate creation of training data and eval data and start eval_state skeleton #89

Closed

6 tasks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

zsusswein zsusswein left review comments

SamuelBrand1 SamuelBrand1 left review comments

damonbayer damonbayer approved these changes

sbidari Awaiting requested review from sbidari sbidari is a code owner

Labels

None yet