Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for large ensembles #127

Open
forsyth2 opened this issue Aug 30, 2021 · 5 comments
Open

Add support for large ensembles #127

forsyth2 opened this issue Aug 30, 2021 · 5 comments
Assignees
Labels
priority: low Low priority task semver: new feature New feature (will increment minor version)

Comments

@forsyth2
Copy link
Collaborator

zppy can currently be run on multiple simulations/members of an ensemble, by using different configuration files to specify different data paths.

It would be nice to add automation such that a single zppy configuration file could be used to post-process all members of an ensemble. Perhaps a summary pdf could be generated as well, displaying mean, median, etc.

@forsyth2 forsyth2 added semver: new feature New feature (will increment minor version) priority: low Low priority task labels Aug 30, 2021
@forsyth2 forsyth2 self-assigned this Aug 30, 2021
@forsyth2 forsyth2 changed the title Improve support for ensembles Add support for large ensembles Aug 30, 2021
@forsyth2
Copy link
Collaborator Author

One potential issue may be submitting/queueing the large number of jobs that would be generated, considering post-processing just one simulation/member launches many jobs.

@forsyth2
Copy link
Collaborator Author

forsyth2 commented Feb 3, 2022

From #186 (a duplicate of this issue): Add support for large ensembles. Add loop to run zppy on multiple experiments.

@wagmanbe
Copy link

wagmanbe commented Feb 4, 2022

I ran zppy on a 250-member ensemble using a code that loops over each case and modifies the .cfg file, then submits the runs if I have less than some threshold number of runs in the queue. It worked great!

@wagmanbe wagmanbe closed this as completed Feb 4, 2022
@rljacob
Copy link
Member

rljacob commented Feb 5, 2022

Is that the solution for all zppy users?

@rljacob rljacob reopened this Feb 5, 2022
@wagmanbe
Copy link

wagmanbe commented Feb 7, 2022

Sorry, I didn't mean to "close" the issue. The method I used works perfectly well, but zppy could certainly come up with its own way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: low Low priority task semver: new feature New feature (will increment minor version)
Projects
None yet
Development

No branches or pull requests

3 participants