Finding the source of UVM error

[Sacusa]

I am running a UVM CUDA application on a NUMA machine. I am trying to use cudaMemAdvise_v2 to initialize the allocation to a DRAM-only NUMA node (i.e., no CPUs). Unfortunately, the call fails with the error: NV_ERR_INVALID_ARGUMENT. Looking at the driver code, my guess is that the problem is in kernel_open/nvidia-uvm/uvm_policy:uvm_api_set_preferred_location, but putting a printk inside the function does not print anything on either the console or /var/log/kern.log (the dmesg level is set to print everything).

I tried following the previous advice on figuring the backtrace, but the first call to nvidia_ioctl has a nv_printf call that does not print anything either. I ensured NVreg_RmMsg is set to print everything.

Any ideas what might be the problem here?

[mtijanic]

Hey there!

but the first call to nvidia_ioctl has a nv_printf call that does not print anything either. I ensured NVreg_RmMsg is set to print everything.

Can you try also setting NVreg_ResmanDebugLevel=0 module param (or change this line to = 0;) and see if that gives you your nv_printf() output?

[Sacusa]

CUDA_ERROR_INVALID_VALUE (0x1) is indeed the error I receive, not NV_ERR_INVALID_ARGUMENT. I assumed that the the API error 0x1 mapped to driver error 0x1f based on my reading of the code.

[mtijanic]

In that case, I'm about 90% sure the issue is that the validation in the CUDA usermode library is failing your code before ever hitting UVM. Maybe post a minimal repro code?

[Sacusa]

Here you go: https://gist.github.com/Sacusa/d7faa2dbe8b2f6b3d23cbbb8907789b2

[Sacusa]

Am I correct in assuming that there is no way to debug the CUDA runtime code? Only the driver code seems to be open source.

[mtijanic]

Sorry, I'm not sure if there's a way to get any logs out of CUDA runtime. Maybe ask on the developer forums that are frequented by the CUDA devs?

I don't see anything immediately wrong with your code, but I hadn't had a chance to run it yet. Do you have HMM enabled?
https://developer.nvidia.com/blog/simplifying-gpu-application-development-with-heterogeneous-memory-management/#enabling_and_detecting_hmm