[RFC] Handle sm_* which is no longer supported by CUDA / ptxas exec check or configure check?

[tob2]

@tschwinge @vries
From https://gcc.gnu.org/pipermail/gcc-patches/2022-March/591154.html

In PR97348, we ran into the problem that recent CUDA dropped support for
sm_30, which inhibited the build when building with CUDA bin in the path,
because the nvptx-tools assembler uses CUDA's ptxas to do ptx verification.
...
Deal with PR97348 in the simplest way possible: when calling the assembler for
sm_30, specify --no-verify.

This has the unfortunate effect that after fixing PR104758 by building
libraries with sm_30, the libraries are no longer verified. This can be
improved upon by:

• adding a configure test in gcc that tests if CUDA supports sm_30, and
  if so disabling this patch
• dealing with this in nvptx-tools somehow, either:
  • detect at ptxas execution time that it doesn't support sm_30, or
  • detect this at nvptx-tool configure time.

[tschwinge]

Another thing that we could do: instead of invoking command-line ptxas for verification, we could dlopen the CUDA Driver (libcuda), and use that one to load and thus verify the PTX code (just like libgomp nvptx plugin and nvptx-tools nvptx-run are doing): https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__MODULE.html, etc.
That way, we could first check which .target values are actually supported by the current CUDA installation, and only adjust if actually necessary. (Well, or probably not at all, since all current versions of the CUDA Driver actually still do load .target sm_30 code?)
We'd avoid using the "narrow" ptxas command-line interface, and instead get the "full" API. (The latter hopefully doesn't bring its own set of compatibility concerns? Not expecting any, given that the libgomp nvptx plugin and nvptx-tools nvptx-run have been stable for a long time in that regard.)

How does the CUDA Driver API differ/relate to the PTX Compiler APIs, https://docs.nvidia.com/cuda/ptx-compiler-api/?

@tob2, @vries

[vries]

Another thing that we could do: instead of invoking command-line ptxas for verification, we could dlopen the CUDA Driver (libcuda), and use that one to load and thus verify the PTX code (just like libgomp nvptx plugin and nvptx-tools nvptx-run are doing): https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__MODULE.html, etc.
That way, we could first check which .target values are actually supported by the current CUDA installation, and only adjust if actually necessary. (Well, or probably not at all, since all current versions of the CUDA Driver actually still do load .target sm_30 code?) We'd avoid using the "narrow" ptxas command-line interface, and instead get the "full" API. (The latter hopefully doesn't bring its own set of compatibility concerns? Not expecting any, given that the libgomp nvptx plugin and nvptx-tools nvptx-run have been stable for a long time in that regard.)

I think I follow your thinking: using the cuda driver api means you're using a defined api rather than the defacto ptxas interface. Note that what you describe checks what .target values are supported by the CUDA driver api, not the CUDA runtime api (in other words, ptxas and friends). Those are separate things.

How does the CUDA Driver API differ/relate to the PTX Compiler APIs, https://docs.nvidia.com/cuda/ptx-compiler-api/?

Ah, did know that one. Well, after I read the introduction part, mainly this looks like the CUDA runtime api equivalent of the CUDA driver api part that does the ptx compilation.

At first glance, it sounds like that part you could use instead of ptxas. But, it's also a recent addition, meaning you'd also require a modern CUDA installation, which kind of could defeat the purpose, or give a very narrow range of CUDAs for which this setup would actually be useful.

Note that it's not a good idea to use the driver API because that requires one to have a driver installed, which doesn't work in a scenario where you use say a server class machine to do heavy toolchain and/or apps builds, and then execute on a separate machine with bulky video cards. We don't want to require (for verification purposes) installing the driver on the server class machine.