You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In the current setup, NVIDIA_VISIBLE_DEVICES env variable is added to configmap so that we pin the pod to a MIG slice. A user pod could have this variable set in the pod at submit time which will provide container access to a slice not chosen by InstaSlice and in the worst case access to all the GPUs on the node. We should modify the webhook to reject such pods at submit time.
The text was updated successfully, but these errors were encountered:
Also, we can consider modifying the webhook to intercept not just pod creation but pod update to make sure users don't intentionally set NVIDIA_VISIBLE_DEVICES to 0.
In the current setup,
NVIDIA_VISIBLE_DEVICES
env variable is added to configmap so that we pin the pod to a MIG slice. A user pod could have this variable set in the pod at submit time which will provide container access to a slice not chosen by InstaSlice and in the worst case access to all the GPUs on the node. We should modify the webhook to reject such pods at submit time.The text was updated successfully, but these errors were encountered: