Skip to content

Commit 39383ab

Browse files
authored
feat: disallow usage of GPUs to CPU-only jobs (#96)
In Nomad jobs that didn't requested GPU, if the job landed in a GPU node and the job's docker image had CUDA libraries installed, then the job could access **all** the GPU of the node. Even if it didn't request any. So for CPU-only jobs we set `NVIDIA_VISIBLE_DEVICES` to `None` to hide GPUs. We also checked that jobs that asked one GPU got indeed only one GPU in the node (Nomad works fine in this case).
1 parent d06d0a6 commit 39383ab

File tree

1 file changed

+1
-0
lines changed

1 file changed

+1
-0
lines changed

ai4papi/routers/v1/deployments/modules.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -283,6 +283,7 @@ def create_deployment(
283283
# Modify the GPU section
284284
if user_conf["hardware"]["gpu_num"] <= 0:
285285
# Delete GPU section in CPU deployments
286+
usertask["Env"]["NVIDIA_VISIBLE_DEVICES"] = "none"
286287
usertask["Resources"]["Devices"] = None
287288
else:
288289
# If gpu_type not provided, remove constraint to GPU model

0 commit comments

Comments
 (0)