Building and Running on SCOREC RHEL7

The following instructions apply to the SCOREC RedHat 7 systems with NVIDIA Turing GPUs. Specific instructions are given for cards with the Turing (e.g., in the blockade system), Pascal (e.g., pachisi, scrabble) and Maxwell architectures.

Setup the environment

Create a file named envGcc10.sh with the following contents:

export root=$PWD # This is the directory containing the kokkos and omega_h repos as subdirs

module unuse /opt/scorec/spack/lmod/linux-rhel7-x86_64/Core 
module use /opt/scorec/spack/v0154_2/lmod/linux-rhel7-x86_64/Core 
module load gcc/10.1.0 mpich
module load cuda/11.4 cmake

function getname() {
  name=$1
  host=`hostname -s`
  buildSuffix=${host}-cuda
  echo "build-${name}-${buildSuffix}"
}
export engpar=$root/`getname engpar`/install # This is where engpar will be (or is) installed
export kk=$root/`getname kokkos`/install   # This is where kokkos will be (or is) installed
export oh=$root/`getname omegah`/install  # This is where omega_h will be (or is) installed
export pumipic=$root/`getname pumipic`/install # This is where PumiPIC will be (or is) installed
export CMAKE_PREFIX_PATH=$engpar:$kk:$oh:$CMAKE_PREFIX_PATH
export MPICH_CXX=$root/kokkos/bin/nvcc_wrapper

source the script to setup your environment in the current shell:

source
envGcc11.sh

Kokkos

** Check your GPU's architecture before running this command! (See note below) **

Build Kokkos with the serial and CUDA backends with debugging (bounds checking) and profiling (integration with kokkos-tools) enabled.

Clone the repo, run cmake, then build and install.

cd $root
[[ ! -d kokkos ]] && git clone [email protected]:kokkos/kokkos.git
mkdir -p $kk
cmake -S kokkos -B ${kk%%install} \
  -DCMAKE_CXX_COMPILER=$root/kokkos/bin/nvcc_wrapper \
  -DKokkos_ARCH_TURING75=ON \
  -DKokkos_ENABLE_SERIAL=ON \
  -DKokkos_ENABLE_OPENMP=off \
  -DKokkos_ENABLE_CUDA=on \
  -DKokkos_ENABLE_CUDA_LAMBDA=on \
  -DKokkos_ENABLE_DEBUG=on \
  -DCMAKE_INSTALL_PREFIX=${kk}
cmake --build ${kk%%install} -j 24 --target install

Note:

For desktop with other types of GPU, the cmake flag, -DKokkos_ARCH_TURING75=ON, could be changed to accommodate the specific GPU architecture.

• Check the GPU type, using nvidia-smi command; the output contains the GPU type.
• Find the corresponding GPU architecture according to this nvidia web page, or the article Matching SM architectures (CUDA arch and CUDA gencode) for various NVIDIA cards.
• Refer to kokkos Architecture Keywords, to set the corresponding cmake flag. For example, GTX970 has a Maxwell architecture with compute capability 5.2, the cmake flag is set as -DKokkos_ARCH_MAXWELL52=ON. Likewise, Quadro P4000, P1000 (scrabble) and the GTX1080ti (Pascal) should be set to -DKokkos_ARCH_PASCAL61=ON.

EnGPar

cd $root
git clone [email protected]:SCOREC/EnGPar.git
mkdir -p $engpar
cmake -S EnGPar -B ${engpar%%install} \
  -DCMAKE_INSTALL_PREFIX=$engpar \
  -DCMAKE_C_COMPILER="mpicc" \
  -DCMAKE_CXX_COMPILER="mpicxx" \
  -DCMAKE_CXX_FLAGS="-std=c++11" \
  -DENABLE_PARMETIS=OFF \
  -DENABLE_PUMI=OFF \
  -DIS_TESTING=OFF
cmake --build ${engpar%%install} --target install -j8 

Note: Loading the pumi module in step one provides all of the dependencies of EnGPar

Omega_h

Setup the environment - the following assumes that the kokkos build environment was setup

** Check your GPU's architecture before running this command! (See note below) **

Clone the repo, run cmake, then build and install.

cd $root
git clone [email protected]:SCOREC/omega_h.git
mkdir -p $oh 
cmake -S omega_h -B ${oh%%install} \
  -DCMAKE_INSTALL_PREFIX=$oh \
  -DBUILD_SHARED_LIBS=OFF \
  -DOmega_h_USE_Kokkos=ON \
  -DOmega_h_USE_CUDA=on \
  -DOmega_h_CUDA_ARCH=61 \
  -DOmega_h_USE_MPI=on  \
  -DBUILD_TESTING=on  \
  -DCMAKE_CXX_COMPILER=`which mpicxx` \
  -DCMAKE_C_COMPILER=`which mpicc` \
  -DKokkos_PREFIX=$kk/lib64/cmake
cmake --build ${oh%%install} --target install -j8

Note:

As explained in Kokkos build section, change the SM cmake flag, -DOmega_h_CUDA_ARCH=75, according to the GPU type if different from above build instruction. You should be able to find the sm flag for the specific GPU in this nvidia web page , where the sm number is the same as compute capability. For example, GTX970 (Maxwell) has a compute capability of 5.2, hence the cmake flag should be set as -DOmega_h_CUDA_ARCH=52. Likewise, Quadro P4000, P1000 and the GTX1080ti (Pascal) should be set to 61. This flag is setting the CMake CUDA_ARCHITECTURES property.

PUMIPic

Clone the repo, run cmake, then build.

cd $root
git clone --recursive [email protected]:SCOREC/pumi-pic.git
[ -d $pumipic ] && rm -rf ${pumipic%%install}
mkdir -p $pumipic
cmake -S pumi-pic -B ${pumipic%%install} \
  -DCMAKE_CXX_COMPILER=mpicxx  \
  -DIS_TESTING=ON  \
  -DPS_IS_TESTING=ON  \
  -DTEST_DATA_DIR=$root/pumi-pic/pumipic-data  \
  -DOmega_h_PREFIX=$oh  \
  -DKokkos_PREFIX=$kk  \
  -DEnGPar_PREFIX=$engpar  \
  -DCMAKE_INSTALL_PREFIX=$pumipic
cmake --build ${pumipic%%install} --target install -j8
ctest --test-dir ${pumipic%%install}

Note:

Switching PS_IS_TESTING to OFF will greatly decrease build and test times. If you are working on particle_structs you'll want to leave it on as these tests are directly exercising particle_structs code.