KREPE (Kernel Replay Execution for Performance Evaluation) captures the execution context of Kokkos kernels and replays them in standalone programs for controlled performance evaluation.
See the documentation and Quickstart for complete guides. Documentation sources are also available in the docs/ directory.
KREPE requires Kokkos 5.0 or newer, a C++20 compiler, CMake 3.26 or newer,
and HDF5 development headers and libraries (C and HL components).
Clone the repo, then follow the installation guide to configure and build with CMake:
git clone https://github.com/CExA-project/KREPE.gitThe library can be included in a CMake project:
find_package(KREPE REQUIRED)
target_link_libraries(my_program PRIVATE KREPE::extractor)The replayed program should link with KREPE::replayer
find_package(KREPE REQUIRED)
target_link_libraries(my_program PRIVATE KREPE::replayer Kokkos::kokkos)In order to extract a kernel from a program, you have to:
- replace the desired
Kokkos::parallel_forwithkrepe::parallel_for, this will allow to save the execution policy and the kernel's data - execute with the
libkrepe.sokokkos tool
For example:
#include <Kokkos_Core.hpp>
#include <krepe/extractor.hpp>
int main(int argc, char* argv[]) {
Kokkos::ScopeGuard kokkos_scope(argc, argv);
const int N = 1024;
Kokkos::View<int*> values("values", N);
Kokkos::parallel_for(
"init", values.size(), KOKKOS_LAMBDA(int i) { values(i) = i; });
// Use KREPE's wrapper to capture this kernel's data and execution policy.
krepe::parallel_for(
"scale", N, KOKKOS_LAMBDA(int i) { values(i) *= 2; });
Kokkos::fence();
return 0;
}The program has to be linked with KREPE::extractor, it then has to be
executed with the following environment variables in order to extract the first
invocation of the kernel named "scale"
KOKKOS_TOOLS_LIBS=/path/to/libkrepe.so \
KOKKOS_TOOLS_ARGS="--krepe-kernel-label=scale --krepe-kernel-invocation=1" \
./progThis generates one HDF5 file named krepe_scale_<kernel_id>.h5.
See the HDF5 file layout
for the file structure and stored metadata.
If you are trying to extract a kernel used in a parallel_for wrapper provided
by another library, you can use the krepe::replay_functor
function to save the functor's data
my_funky_parallel_for("kernel", N,
krepe::replay_functor(
KOKKOS_LAMBDA(int i) { values(i) *= 2; }));Note that the execution policy cannot be saved in this case.
By default, if the compiler supports it, we build compiler plugins which allow us to have fine-grained control over which allocations are dumped (instead of dumping every allocation up to the current kernel launch). They work by introspecting the functor to search for Views contained inside of it, allowing to only export allocations which correspond to one of the functor's views. Plugins are currently supported for GCC, NVCC using GCC as host compiler, Clang and Clang-based compilers (e.g. hipcc).
If needed, compiler plugins can be disabled with -DKREPE_ENABLE_COMPILER_PLUGINS=OFF.
See the compiler plugin setup for GCC and Clang requirements.
Once the program dump has been generated, the kernel can be replayed in a
separate program. The new program should include the parallel construct call as
well as the functor declaration from the original program and any variable it
depends on. The replayer should also be initialized before Kokkos, using
krepe::ScopeGuard.
The program above becomes:
#include <Kokkos_Core.hpp>
#include <krepe/replayer.hpp>
int main(int argc, char* argv[]) {
krepe::ScopeGuard replay_scope(argc, argv);
Kokkos::ScopeGuard kokkos_scope(argc, argv);
Kokkos::View<int*> values;
// Replay with the captured inputs and policy (the 0 argument is ignored).
krepe::parallel_for(
"scale", 0, KOKKOS_LAMBDA(int i) { values(i) *= 2; });
Kokkos::fence();
return 0;
}The program has to be linked with KREPE::replayer. Pass the dump using
the --krepe-dump-file command line flag:
./replay_prog --krepe-dump-file="krepe_scale_<kernel_id>.h5"Replace krepe_scale_<kernel_id>.h5 with the filename reported during capture.
Complete capture and
replay examples are available in the repository.
Enable --krepe-enable-input-reset, then call krepe::reset_inputs()
before each replay to restore the captured inputs.
See Benchmarking & auto-tuning
for the input reset option and a replay loop example.
By default, krepe::parallel_for will use the execution policy
that was saved in the dump. You can override this by passing
krepe::force_policy(your_policy) as the execution policy
argument.
krepe::get_allocation returns the active replay buffer, while
krepe::get_out_allocation returns the reference output recorded during capture.
See Accessing the allocations
for an example using unmanaged Views.
See Limitations for capture and replay restrictions.
The full license statement used in all headers is available here.