Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Snakemake resources does not trigger GPU node-pool on kubernetes #30

Open
mrburke00 opened this issue Nov 14, 2024 · 0 comments
Open

Snakemake resources does not trigger GPU node-pool on kubernetes #30

mrburke00 opened this issue Nov 14, 2024 · 0 comments

Comments

@mrburke00
Copy link

Using snakemake/snakemake:v8.20.3 for Kubernetes jobs.

I am trying to set GPU resources for a rule to execute on GKE, but the GPU node pool is never triggered.

snakemake --snakefile workflow/snakefile_stage3_totalvi.smk \
          --executor kubernetes \
          --software-deployment-method conda \
          --default-storage-provider gcs \
          --default-storage-prefix $BUCKET \
          --storage-gcs-project $PROJECT \
          --set-resources totalvi_train.smk:gpu=1 \
          --jobs 1

Ive also tried resources gpu=1 and default-resources gpu=1 as well as setting the resources gpu=1 directly in the rule itself.

Changing the mem and cores resources does in fact trigger the respective node pools so the problem is gpu specific.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mrburke00