iGUARD: In-GPU Advanced Race Detection

We provide the source code and the setup for iGUARD, a tool to detect races in GPU programs. iGUARD instruments GPU programs to detect races in them. It uses NVIDIA's NVBit [1], a GPU binary instrumenter, as the framework for instrumentation.

NOTE: This branch uses an updated version of NVBit and has not been extensively tested. Use the artifact-eval branch for the version we used in our paper.

This README provides a peek into different parameters of the tool and a very high-level view of source code organization.

For full details refer to our paper: [Paper] [Video]

iGUARD: In-GPU Advanced Race Detection
Aditya K Kamath and Arkaprava Basu
ACM SIGOPS 28th Symposium on Operating Systems Principles (SOSP), 2021
DOI: https://doi.org/10.1145/3477132.3483545

Benchmarks

To replicate the results given in the paper, we provide precompiled application binaries. Full details are given in the README in the benchmarks folder.

Hardware and software requirements

iGUARD is built on top of NVBit (version 1.7.4) and shares its requirements, listed below:

• SM compute capability: >= 3.5 && <= 9.2
• Host CPU: x86_64, aarch64
• OS: Linux
• GCC version: >= 5.3.0 for x86_64; >= 7.4.0 for aarch64
• CUDA version: >= 12.0
• CUDA driver version: <= 555.xx
• nvcc version for tool compilation >= 10.2

Because NVBit does not require application source code, any pre-compiled GPU application should work regardless of which compiler (or version) has been used (i.e. nvcc, pgicc, etc).

Required software packages can be installed through apt using the following command:

sudo apt-get install -y wget bc gcc time gawk libtbb-dev

Behind the scenes: iGUARD's parameters, setup, and source code

Compilation

To download NVBit and compile the detector, run the following command:

make setup

The compiled detector can be found at nvbit_release/tools/detector/detector.so.

Running the race detector (iGUARD)

Once compiled, the detector can be run on binaries containing NVIDIA GPU code by setting the LD_PRELOAD environment variable. For example, to run the detector on an application binary called app.exe contained in the main repository folder, you would run the following command:

LD_PRELOAD=./nvbit_release/tools/detector/detector.so ./app.exe

Compiling the application binary with -lineinfo flag allows iGUARD to output line numbers when races are detected. Otherwise, SASS offsets are used.

Race detection options (parameters)

We provide several knobs that allow users to change how the detection works. The major ones are listed below:

• BYTE_GRAN: The granularity of a single data item (in bytes) considered by the detector. (default = 4)
• CONT_OPT: Whether to perform contention optimizations (exponential backoff and detection coalescing) during race detection. (default = 1)
• EXIT: Whether to quit on encountering the first race. (default = 0)
• TIMEOUT: Time-out in seconds after which the application is terminated. 0 means never. (default = 0)

To use these knobs, set them as environment variables when performing race detection. For example, to time-out after 4 seconds when performing detection on app.exe, we run the following command:

TIMEOUT=4 LD_PRELOAD=./nvbit_release/tools/detetor/detector.so ./app.exe

Source code

The source code for the iGUARD race detector is found in the detector/ folder. The major files are as follows:

• detector.cu: This contains the CPU-side code for the detector. This includes allocating memory for metadata, the binary instrumentation process, and outputting caught races to the user.
• inject_funcs.cu: This contains the CUDA code run on the GPU after instrumentation. This includes incrementing relevant counters on synchronization operations, adding/removing locks from the lock table, and performing the in-GPU race detection.

References

[1] NVBit [Paper] [Repository]