CPU Build

Setup the environment

Create a file named envGcc10.sh with the following contents:

export root=$PWD # This is the directory containing the kokkos and omega_h repos as subdirs

module unuse /opt/scorec/spack/lmod/linux-rhel7-x86_64/Core 
module use /opt/scorec/spack/v0154_2/lmod/linux-rhel7-x86_64/Core 
module load gcc/10.1.0 mpich
module load cuda/11.4 cmake

function getname() {
  name=$1
  host=`hostname -s`
  buildSuffix=${host}-serial
  echo "build-${name}-${buildSuffix}"
}
export engpar=$root/`getname engpar`/install # This is where engpar will be (or is) installed
export kk=$root/`getname kokkos`/install   # This is where kokkos will be (or is) installed
export oh=$root/`getname omegah`/install  # This is where omega_h will be (or is) installed
export pumipic=$root/`getname pumipic`/install # This is where PumiPIC will be (or is) installed
export CMAKE_PREFIX_PATH=$engpar:$kk:$oh:$CMAKE_PREFIX_PATH
export MPICH_CXX=`which g++`

source the script to setup your environment in the current shell:

source
envGcc11.sh

Kokkos

Build Kokkos with the serial and CUDA backends with debugging (bounds checking) and profiling (integration with kokkos-tools) enabled.

Clone the repo, run cmake, then build and install.

cd $root
[[ ! -d kokkos ]] && git clone [email protected]:kokkos/kokkos.git
mkdir -p $kk
cmake -S kokkos -B ${kk%%install} \
  -DCMAKE_CXX_COMPILER=$MPICH_CXX \
  -DKokkos_ENABLE_SERIAL=ON \
  -DKokkos_ENABLE_OPENMP=off \
  -DKokkos_ENABLE_CUDA=off \
  -DKokkos_ENABLE_CUDA_LAMBDA=off \
  -DKokkos_ENABLE_DEBUG=on \
  -DCMAKE_INSTALL_PREFIX=${kk}
cmake --build ${kk%%install} -j24 --target install

EnGPar

cd $root
git clone [email protected]:SCOREC/EnGPar.git
mkdir -p $engpar
cmake -S EnGPar -B ${engpar%%install} \
  -DCMAKE_INSTALL_PREFIX=$engpar \
  -DCMAKE_C_COMPILER="mpicc" \
  -DCMAKE_CXX_COMPILER="mpicxx" \
  -DCMAKE_CXX_FLAGS="-std=c++11" \
  -DENABLE_PARMETIS=OFF \
  -DENABLE_PUMI=OFF \
  -DIS_TESTING=OFF
cmake --build ${engpar%%install} --target install -j8 

Note: Loading the pumi module in step one provides all of the dependencies of EnGPar

Omega_h

Setup the environment - the following assumes that the kokkos build environment was setup

Clone the repo, run cmake, then build and install.

cd $root
git clone [email protected]:SCOREC/omega_h.git
mkdir -p $oh 
cmake -S omega_h -B ${oh%%install} \
  -DCMAKE_INSTALL_PREFIX=$oh \
  -DBUILD_SHARED_LIBS=OFF \
  -DOmega_h_USE_Kokkos=ON \
  -DOmega_h_USE_CUDA=off \
  -DOmega_h_USE_MPI=on  \
  -DBUILD_TESTING=on  \
  -DCMAKE_CXX_COMPILER=`which mpicxx` \
  -DCMAKE_C_COMPILER=`which mpicc` \
  -DKokkos_PREFIX=$kk/lib64/cmake
cmake --build ${oh%%install} --target install -j8

PUMIPic

Clone the repo, run cmake, then build.

cd $root
git clone --recursive [email protected]:SCOREC/pumi-pic.git
[ -d $pumipic ] && rm -rf ${pumipic%%install}
mkdir -p $pumipic
cmake -S pumi-pic -B ${pumipic%%install} \
  -DCMAKE_CXX_COMPILER=mpicxx  \
  -DIS_TESTING=ON  \
  -DPS_IS_TESTING=ON  \
  -DTEST_DATA_DIR=$root/pumi-pic/pumipic-data  \
  -DOmega_h_PREFIX=$oh  \
  -DKokkos_PREFIX=$kk  \
  -DEnGPar_PREFIX=$engpar  \
  -DCMAKE_INSTALL_PREFIX=$pumipic
cmake --build ${pumipic%%install} --target install -j8
ctest --test-dir ${pumipic%%install}

Note:

Switching PS_IS_TESTING to OFF will greatly decrease build and test times. If you are working on particle_structs you'll want to leave it on as these tests are directly exercising particle_structs code.