Using GPU functionality with a custom docker image

[ashwinjosephk]

Hi Community ,
I am new to the world of Dockers and would appreciate help to help me resolve an issue I am facing!

I tried your example here with ubuntu docker image it detects the nvidia drivers:

I am trying to setup a docker instance for me to carry out a calibration process. The command have been provided by the author too but when I try to execute it, apparantly GPU is not getting utilised though it is installed in the host system. The repo is:
https://github.com/koide3/direct_visual_lidar_calibration/tree/main

I am trying to run the calibration on the environemnt:ROS1 and Ubuntu 20
My commands are as follows:

# Define paths
bag_path=$(realpath /home/knadmin/Ashwin/Targetless_calib/livox_ros1)
preprocessed_path=$(realpath /home/knadmin/Ashwin/Targetless_calib/livox_ros1_preprocessed)

docker run \
  -it \
  --net host \
  --runtime=nvidia --gpus all  \
  -e DISPLAY=$DISPLAY \
  -v $HOME/.Xauthority:/root/.Xauthority \
  -v $bag_path:/tmp/input_bags \
  -v $preprocessed_path:/tmp/preprocessed \
  -v pip_cache:/root/.cache/pip \
  koide3/direct_visual_lidar_calibration:noetic /bin/bash nvidia-smi

but apparenty nvidia-smi is not getting recognized.

What should i do to rectify this issue? I understand this might be slightly off topic but I would be really helpful to udnerstand what is going wrong and how I can fix it. I checked the dockefile its already usign another required base image.

[yeongrokgim]

As a quick debug, try append following environment variables so that nvidia-container-runtime would inject GPU related resources.

@@ -1,10 +1,12 @@
 docker run \
   -it \
   --net host \
   --runtime=nvidia --gpus all  \
   -e DISPLAY=$DISPLAY \
+  -e NVIDIA_VISIBLE_DEVICES=all \
+  -e NVIDIA_DRIVER_CAPABILITIES=all \
   -v $HOME/.Xauthority:/root/.Xauthority \
   -v $bag_path:/tmp/input_bags \
   -v $preprocessed_path:/tmp/preprocessed \
   -v pip_cache:/root/.cache/pip \
   koide3/direct_visual_lidar_calibration:noetic /bin/bash nvidia-smi

[elezar]

@ashwinjoseph95 could it be that the image that you're trying to run does not have the path set up such that nvidia-smi injected by the NVIDIA Continer Toolkit can be located.

What is the output of the env command in the container? Whats happens when you run /usr/bin/nvidia-smi instead of nvidia-smi?

@yeongrokgim thanks for the tip. Note that the --gpus all flag should be setting NVIDIA_VISIBLE_DEVICES=all in the docker daemon.