Skip to content

Separate CDS request logic from other preprocessing #26

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
trossi opened this issue Jan 29, 2025 · 1 comment
Open

Separate CDS request logic from other preprocessing #26

trossi opened this issue Jan 29, 2025 · 1 comment

Comments

@trossi
Copy link
Collaborator

trossi commented Jan 29, 2025

A lot of runtime in ModGP-run_prep.R is spent idling and waiting CDS to process requests. Other processing in the script requires more resources (e.g. memory), so it would be useful to separate the waiting/downloading logic from the actual processing.

This could be implemented as a separate initial script that does only the CDS requests and downloads all necessary Data/Environment/TEMP_* files that are then processed by ModGP-run_prep.R further as currently.

See related discussion in #25 and related issues #33 #35 #37.

@MichalTorma
Copy link
Collaborator

I think this is a good idea but not hight on a priority list. It's only a little wasteful - ideally data prep like this should be done outside LUMI completely - since it doesn't need much processing anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants