Replies: 52 comments 124 replies
-
|
Hello and thank you for your interest! RM stands for "Resource Manager" and it is the internal name of the driver that gets compiled into Depending on the context, GSP-RM can refer either to the new driver architecture that uses the "GSP" microcontroller, or it can refer to the code running on said microcontroller. To avoid ambiguity, the latter is often called "Physical RM" instead, and code running in kernel is called "Kernel RM". RMAPI is the interface nvidia.ko (RM) exposes to other kernel modules and to userspace. |
Beta Was this translation helpful? Give feedback.
-
|
Meta: Do you think it makes sense to group these questions into a single issue, which we could then pin? |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! Yes, I think it's a good idea to group these questions into a single issue. |
Beta Was this translation helpful? Give feedback.
-
|
Okay, this is now a pinned generic code-related Q&A issue. |
Beta Was this translation helpful? Give feedback.
-
|
Would it be possible to enable Github Discussions, which has a cleaner structure for quick questions? |
Beta Was this translation helpful? Give feedback.
-
|
@antdking Please see #44 . I do agree Discussions would be a much cleaner option, and I expect we'll enable them soon enough. Will gladly retire this issue when that happens, but until such a time this is the best we have for these quick one-off questions. |
Beta Was this translation helpful? Give feedback.
-
|
Will this driver support GPU virtualization with untrusted guests on consumer hardware? This would be a huge win for Qubes OS, which is currently forced to rely on software rendering. Support for CUDA or for enterprise GPUs would not be as helpful, as Qubes OS is an end-user operating system focused on the desktop use-case. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @DemiMarie , thanks for the interest. The currently published driver does not support virtualization, neither as a host nor a guest. I currently don't have any roadmap information to share regarding this. |
Beta Was this translation helpful? Give feedback.
-
|
@DemiMarie At this point all I can say is that the current codebase does not support virtualization - neither as a host nor a guest. I don't currently have any information about future changes in this regard. |
Beta Was this translation helpful? Give feedback.
-
|
We've now enabled Discussions in this github repository. I've moved this issue to Discussions, though maybe we should close this and start a separate discussion for each question... |
Beta Was this translation helpful? Give feedback.
-
|
As many of us have noticed so far. NVIDIA has defaulted to one of the age old standards of C90. Moving forward, would NVIDIA be open to adopting more modern standards such as C11? or is there a very specific reason as to why we must stay C90? My argument for adopting the newer standard would be the newer features, most prominently seen in C++ via operators, templates, and constexpr. constexpr in particular leads to less hardcoded variables in favor of a compile time resolved one. What do you think? |
Beta Was this translation helpful? Give feedback.
-
|
It would be important for the standard to be upgraded to c11. |
Beta Was this translation helpful? Give feedback.
-
|
At least it's not C89. I'll submit a draft PR for this. I'm sure NVIDIA knows about C11 though; it's probably because they need these kernel modules to be able to be developed on any platform. On second thought, C11 doesn't have many killer features. C99 should be the next logical step from C90, because the difference between C90 and C99 is huge and would be most desirable for those aforementioned compatibility considerations and for modern features. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for wanting to bring us into the 21st century! Unfortunately, much of the code published here is shared by various other projects and is compiled for a wide variety of platforms - from Windows to custom NV-internal ISAs and OSes - which means it must support a wide variety of toolchains and their quirks. Generally, this means that we need to stick with C90. Many (but not all) C99 features are supported, but in some cases they are avoided due to style guidelines. We hope to publish a version of these guidelines in the future. |
Beta Was this translation helpful? Give feedback.
-
|
I'm curious what the reason was for retaining whitespace in what look like deleted sections of the source eg https://github.com/NVIDIA/open-gpu-kernel-modules/blob/main/kernel-open/nvidia/nv.c#L623. Was this intentional or just a side effect on how it was done? |
Beta Was this translation helpful? Give feedback.
-
|
Likely just a side effect of how they chose to get rid of the lines and nothing else. |
Beta Was this translation helpful? Give feedback.
-
|
You can see it here: |
Beta Was this translation helpful? Give feedback.
-
|
This is an issue with our packaging scripts that removed parts of the codebase that are not relevant to this driver. Successive blank lines should have been collapsed into a single one. We will fix the scripts. |
Beta Was this translation helpful? Give feedback.
-
|
Considering that NVIDIA does not currently have plans to open-source the userspace components of the driver, at the minimum, will there be any officiial documentation or libraries to help create a userspace RM client (I think this constitutes opening and controlling |
Beta Was this translation helpful? Give feedback.
-
|
This is good news, no rush on the documentation. I look forward to writing my first RM client one day hopefully in the near future! |
Beta Was this translation helpful? Give feedback.
-
In that case, would it be possible to provide an open-source userspace that does have a stable API and ABI? |
Beta Was this translation helpful? Give feedback.
-
|
There are currently no plans to open source NVML or NVAPI, nor to maintain a third library, sorry. |
Beta Was this translation helpful? Give feedback.
-
|
That’s unfortunate. Such a library really ought to be in this repository, so that it can be kept in sync with kernel driver changes. |
Beta Was this translation helpful? Give feedback.
-
|
I am also interested in writing a userspace RM client. Is there any documentation or example code that might help me accomplish this? I am aware that there is no ABI stability and I'm willing to target a particular driver version. |
Beta Was this translation helpful? Give feedback.
-
|
Since the other topic got locked Ill ask here too. While this Repo provides kernel drivers for turing, ampere and upcoming generations will PMU firmware be published to allow nouveau to reclock Maxwell 2 and Pascal GPUs? |
Beta Was this translation helpful? Give feedback.
-
|
Skimming through the codebase, I see references to something called |
Beta Was this translation helpful? Give feedback.
-
|
See here for NVOC.
Almost everything starts with "NV" or "RM", so you can safely just treat those as namespace and not a meaningful part of the name :) |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the information! (Should have looked around more…) |
Beta Was this translation helpful? Give feedback.
This comment was marked as off-topic.
This comment was marked as off-topic.
-
|
Does the kernel driver handle stub generation for CUDA binaries that use host functions (i.e. printf)? I noticed that when I generate the SASS binary, the relocation isn't handled even though it's technically "executable". Does CUDA take care of the relocaiton for me? When I dump the CUDA binary after it's been copied to the GPU, it is filled in for me. How does it know which address to fill it in with? How can I control this aspect? |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I have the same question, have you figured it out? |
Beta Was this translation helpful? Give feedback.
-
|
What are the effects of NVC6C0_INVALIDATE_SHADER_CACHES_NO_WFI and NVC6C0_SET_REPORT_SEMAPHORE_A? |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I'm monitoring the ioctl commands for a hw accelerated video encoding session. I see a series of calls to C9B7 (NVC9B7_VIDEO_ENCODER) class. However, class header here https://github.com/NVIDIA/open-gpu-kernel-modules/blob/main/src/common/sdk/nvidia/inc/class/clc9b7.h is empty. I found a similar header for C5B7 here: https://github.com/NVIDIA/open-gpu-doc/blob/master/classes/video/clc5b7.h and the command codes seem to match, but I wonder if someone can confirm that the codes are the same, or even better, fill the header for class C9B7. Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
What is "FINN, an NVIDIA coding tool"? It it some kind of IDL compiler for RM API to serialize API calls into binary streams? I am currently attempting to run Nvidia driver as userland server process in microkernel-like architecture, so I need to serialize RM API calls into IPC messages. Is |
Beta Was this translation helpful? Give feedback.
-
|
Yes, FINN is a serializer/deserializer; it is still a work-in-progress. Thus far, we've not published the finn tool itself, just the code generated by finn. I'm not sure rmapi_finn.c has all the information it would need to serialize all the RMAPIs, but you're certainly welcome to use it as a starting point. Our QNX support uses a separate code base, and not the resource manager code from which the open-gpu-kernel-modules is derived. Good luck! |
Beta Was this translation helpful? Give feedback.
-
|
This is how far I got, it seems running driver is userland is possible, but UNIX API may not fit well (no FDs etc.). It is developed for Haiku operating system. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
While running TensorFlow inference tasks with Docker, I noticed that when tracking the code, 16 channels are always created during task execution:
Is it possible for
|
Beta Was this translation helpful? Give feedback.
-
|
GPU: nvidia A10 cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX Open Kernel Module for x86_64 535.54.03 Release Build (xxxxx) Mon 25 Mar 2024 02:39:25 PM HKT
GCC version: gcc version 10.2.1 20210110 (Debian 10.2.1-6) |
Beta Was this translation helpful? Give feedback.
-
|
After some break on Nvidia driver porting experiments it started to fail to halt Falcon, but it worked before. Any ideas what is happening and how to fix it? ACPI OS interface is not implemented yet. Both versions 545.29.02 and 550.90.07 fail in the same way. UEFI boot is used. Log |
Beta Was this translation helpful? Give feedback.
-
|
I'm sorry, I don't have any ideas off hand. Does the failure come while trying to initialize GSP, or when trying to shutdown? Assuming this is during initialization, the best I can think to suggest is to step-by-step compare the GSP bootstrap sequence on Linux with what happens in your Haiku port. Out of curiosity: what is all different in your port? Have you just had to provide an alternate implementation of everything under kernel-open/nvidia/..., or have you had to make more substantial changes to the core of the code base? Knowing what is different might help identify where to look for problems that could explain the failure you are seeing. |
Beta Was this translation helpful? Give feedback.
-
|
No changes are made to platform-independent Nvidia driver part except Haiku build fixes and logging. Result object file |
Beta Was this translation helpful? Give feedback.
-
|
Hi! Could anyone please tell what the abbreviations QMD, PCAS and SKED mean? |
Beta Was this translation helpful? Give feedback.
-
|
QMD is "Queue MetaData", but it's a very confusing name. It's a hardware defined structure that contains information to execute some compute work. We also sometimes just call it a "task". SKED is a HW engine that schedules QMDs (it's not an acronym). PCAS is "Posted Compare And Swap" (I had to look it up). It's basically an equivalent of |
Beta Was this translation helpful? Give feedback.
-
|
Also what is the meaning of "WAR" found in various comments? Example: |
Beta Was this translation helpful? Give feedback.
-
|
WAR = Workaround. Usually software workarounds for certain silicon bugs that won't be fixed in hardware, but we sometimes use the term in case of a temporary hacky fix that we plan to improve in future too. |
Beta Was this translation helpful? Give feedback.
-
|
"WAR" is commonly used within NVIDIA as an abbreviation for "Work ARound". |
Beta Was this translation helpful? Give feedback.
-
|
@mtijanic Hi, I noticed that you mentioned a few years ago that you could provide sample input and output files of NVOC (link). I'm very curious about this, but I can't find it in any public place. Is it convenient for you to provide it at this moment? Thank you so much! |
Beta Was this translation helpful? Give feedback.
-
|
(sorry, was out of office) We actually shipped a few NVOC files like RsClient or CeUtilsApi that roughly show the syntax and stuff. You can look at the There's a bit of extra syntax sugar around to let us define different behavior per chip. For example, you can search for |
Beta Was this translation helpful? Give feedback.
-
|
can i have a detailed documentation of the source code... a module , file wise documentation |
Beta Was this translation helpful? Give feedback.
-
|
Me too! (Sorry, we don't currently have any such doc that is suitable for public use. Maybe in the future someone can sit down and write something, but so far it has been pretty low on the priorities list) |
Beta Was this translation helpful? Give feedback.
-
|
Do NVRM API have some engine load ratio, temperature and fan speed monitoring functionality? Information that is used for example in Windows Task Manager GPU load tab. |
Beta Was this translation helpful? Give feedback.
-
|
RM generally exposes it through this memory structure that is then mapped to userspace: https://github.com/NVIDIA/open-gpu-kernel-modules/blob/main/src/common/sdk/nvidia/inc/class/cl00de.h#L244-L298 However, you're advised to not use that directly as it changes every driver release and your app would need to be tightly coupled to the driver. Instead, use NVML or NVAPI (or the X driver XNVCTRL). These are userspace libraries that are shipped with the driver and wrap the RMAPI in a way that is stable across versions. |
Beta Was this translation helpful? Give feedback.
-
|
Proprietary components are not an option. |
Beta Was this translation helpful? Give feedback.
-
|
Is it possible to get per-engine load? |
Beta Was this translation helpful? Give feedback.
-
|
The various video engines load is available individually: https://github.com/NVIDIA/open-gpu-kernel-modules/blob/main/src/common/sdk/nvidia/inc/class/cl00de.h#L136-L146 Keep in mind that this interface is pretty new and is evolving rather fast still, so keep an eye out on changes with every major version. You can also get the utilization info with NV2080_CTRL_CMD_PERF_GET_GPUMON_PERFMON_UTIL_SAMPLES_V2 but then you are actually synchronously fetching the data from GSP and blocking GSP from doing other work, so I don't recommend doing this for monitoring, or you'll get stutter. |
Beta Was this translation helpful? Give feedback.
-
|
What is meaning of LTP? (with uTLB) |
Beta Was this translation helpful? Give feedback.
-
|
L1, TEX, PE (Primitive Engine). |
Beta Was this translation helpful? Give feedback.
-
|
A couple questions regarding module options:
I guess the million dollar question is whether these actually do anything or (for example) they're actually enabled by default on modern gpus (maybe inside the GSP?) |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
How GPU fetch the push buffer? by DMA or others? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, that's correct. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, @aaronp24 |
Beta Was this translation helpful? Give feedback.
-
|
It depends on what the CPU is looking for. If it just wants to see if there's room for more data in the push buffer or the GPFIFO, it can read the There other types of progress tracking indicators available as well. See, for example, the |
Beta Was this translation helpful? Give feedback.
-
|
Can the driver intercept a kernel launch or other high-level operations such as context creation or module loading? |
Beta Was this translation helpful? Give feedback.
-
|
Something like cuCtxCreate() involves dozens of separate calls into the kernel driver. You can certainly catch any one of them and poke around. You can also try to do this in userspace by overriding ioctl(), or by using strace, or something like bpftrace. However, actual kernel launch and stuff is mostly handled directly from userspace to GPU via shared memory, and the kernel is not involved. To instrument that, I believe envytools has something based on userfaultfd that will log all the writes to this memory that the UMD makes. |
Beta Was this translation helpful? Give feedback.
-
|
Here's a list of all the syscalls a CUDA app runs as part of cuInit() + cuCtxCreate(): https://gist.github.com/mtijanic/aabdfd00d9c73491c74638da826ed6d4 (gathered with bpftrace) |
Beta Was this translation helpful? Give feedback.
-
|
Can we schedule channels by Env: Tesla T4 with open source driver 550.90.07. Here is what I found:
It seems that I can disable a channel but could not enable it again to let CUDA kernels go on. Am I wrong to use this command? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @mtijanic @aaronp24 @gauravjuvekar, I'm tring to enable and disable the channel every 2s, but I found sometimes the cuda kernel will be stuck and the GPU utilization decreases to 0. Do you have any ideas about this? |
Beta Was this translation helpful? Give feedback.
and others
-
[mtijanic] Please use this issue to ask simple questions about the codebase, such as what a certain feature is, how a given codepath works, what a given acronym means, etc. Please keep your questions as specific as possible.
I see a lot of RM in the code related to the RMAPI module, what does this RM and GSP-RM mean exactly?
Beta Was this translation helpful? Give feedback.
All reactions