Why the "OS-agnostic" driver function cannot be probed

[PGKANG]

When I trace the nv driver OS-agnostic function "rmapiFreeWithSecInfo" by eBPF tools(command: sudo ./bpftrace -e 'kprobe:rmapiFreeWithSecInfo { print(kstack); }'), I got the messages as below:

stdin:1:1-28: WARNING: rmapiFreeWithSecInfo is not traceable (either non-existing, inlined, or marked as "notrace"); attaching to it will likely fail
kprobe:rmapiFreeWithSecInfo { print(kstack); }
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Attaching 1 probe...
cannot attach kprobe, Invalid argument
ERROR: Error attaching probe: 'kprobe:rmapiFreeWithSecInfo'

Is anyone can help with this? How to build the driver with "probe" enabled so that I would debug the nvidia driver issue?

[mtijanic]

Hey there, I think something like this should make it easier to profile:

diff --git a/src/nvidia/Makefile b/src/nvidia/Makefile
index 423414af..22f98fea 100644
--- a/src/nvidia/Makefile
+++ b/src/nvidia/Makefile
@@ -151,6 +151,12 @@ CFLAGS += -fdata-sections
 NV_KERNEL_O_LDFLAGS += --gc-sections
 EXPORTS_LINK_COMMAND = exports_link_command.txt
 
+CFLAGS += -pg
+CFLAGS += -mfentry
+CFLAGS += -mrecord-mcount
+CFLAGS += -fno-omit-frame-pointer
+CFLAGS += -fno-optimize-sibling-calls
+
 ifeq ($(TARGET_ARCH),x86_64)
   COMPILER_TYPE := $(call GET_COMPILER_TYPE, $(CC))
   ENDBR_SUPPORTED := $(call AS_HAS_INSTR, endbr64)
diff --git a/src/nvidia/nv-kernel.ld b/src/nvidia/nv-kernel.ld
index 89ce3669..e525ec26 100644
--- a/src/nvidia/nv-kernel.ld
+++ b/src/nvidia/nv-kernel.ld
@@ -30,6 +30,8 @@ SECTIONS {
 
   .bss    : { *(.bss) *(.bss.*) }
 
+  __mcount_loc : { KEEP(*(__mcount_loc)) }
+
   /* The rest of the sections ("orphaned sections") will just be copied from
      the input to the output */
 }

Please let me know if it works or not.

[mtijanic]

Can you give me a few examples of the functions that can't be traced with the patch?

By the way, is this for a production system, or a scratch dev system where you don't care about stability and just want to play around?

[PGKANG]

It is ubuntu 22.10 release version. And here is an example:

1. rmapiControlWithSecInfo can be traced
2. rmapiAllocWithSecInfo cannot be traced

And I find another thing, these funcitons cannot be traced by ftrace, but can be traced by bpftrace.

[mtijanic]

Well, that explains why I never hit this. I only ever use bpftrace :)

I'll check internally with some folks and get back to you.

[PGKANG]

Ok, thanks.

[PGKANG]

Hi, I try your previous patch in another environment, it wroks fine. I think your previous patch has indeed solved the problem, thanks for the answer.