Basic example doesn't quite work

[carstenbauer]

• https://github.com/ROCm-Developer-Tools/roctracer/blob/4f97ea2c4427887f5718760e9d315d1f776c5eeb/test/app/roctx_test.cpp
• https://github.com/ROCm-Developer-Tools/roctracer/blob/amd-master/test/hip/MatrixTranspose.cpp

[carstenbauer]

See https://github.com/JuliaGPU/ROCTX.jl/tree/main/examples/roctx_test. The Julia variant runs just fine but doesn't seem to work, i.e., no marker/ranges sections appear in the results.json.

Note that a virtually identical C++ version produces the expected output on the same system.

[carstenbauer]

FWIW, adding ROCTX.roctracer_start() and ROCTX.roctracer_stop() doesn't seem to change anything.

[luraess]

@matinraayai - may this be related to some LD_PRELOAD behaviour?

[carstenbauer]

FWIW, I tried setting LD_PRELOAD similar to what they do for the original C++ example. Same result/problem.

[carstenbauer]

FWIW, a colleague told me that he saw something similar (no traces in the output file) with C++ and MPI. He suggested I should try omniperf omnitrace.

[carstenbauer]

Update on omnitrace:

omnitrace-instrument works fine for the simple C++ example (logfile: omnitrace_Cpp.txt):

Doesn't work for Julia 😢 (logfile: omnitrace_Julia.txt). The critical line in the log is probably:

omnitrace-instrument: /home/omnitrace/external/dyninst/dyninstAPI/src/BPatch.C:752: void BPatch::registerForkedProcess(PCProcess*, PCProcess*): Assertion `parent' failed.

(cc @vchuravy, not because I think you'll work on this much but maybe you can translate "complicated log file" → "this is the problem" statement 😄 )

[matinraayai]

@carstenbauer I've never used roctx, but I've written my own AMD tool to capture HSA/HIP APIs, and it worked fine for @luraess's issue.
I'm suspecting some shenanigans going on when rocprof dlsyms the roctx callback register function which gets hidden from Julia. Unfortunately there's not a good logging mechanism going on to roctracer so I have to build roctx/rocprof with debug information to confirm that is indeed the case. I will get to it probably by the end of this week.

[matinraayai]

@carstenbauer just as I suspected in the Julia version the RocTXLoader's dlsym handle is not loaded and returns nullptr. This means that the RocTX's callbacks are not registered in the first place, making your calls to the library (which are called correctly BTW) ineffective, even though your are storing a callback function for RocTX.
To give more background, rocprof itself is just a bash script setting environment variables to the following before launching your executable. In our case, they are:
HSA_TOOLS_LIB=/opt/rocm/lib/libroctracer64.so /opt/rocm/lib/libroctracer_tool.so
ROCTRACER_DOMAIN=roctx

For each tool loaded at startup, roctracer tool library provides the HSA API table to capture them and replace them with whatever they want, and do additional initialization. During this initialization process, the libroctracer64.so library is dlsymed by Roctracer to install different callbacks in different domains (HIP, HSA, RocTX, etc).

This work without any issue for C++, but doesn't work for Julia. The exact place that this happens is here.

I think Julia's load of this library for ccalls is interfering somehow with the library's intended dlsym. @vchuravy @carstenbauer thoughts?