Error when running in parallel on IBM system

[mulligatawny]

The turbPipePeriodic example works for me when I run in serial, but I see this error

calling nek_userchk ...

generating t-mesh ...
loading mesh from nek ... NboundaryIDs: 1, NboundaryFaces: 47040 done (0.000474442s)
N: 7, Nq: 8, cubNq: 11
computing geometric factors ... J [3.12572e-06,0.00019353] done (2.18544s)
meshParallelGatherScatterSetup N=7
timing gs modes: 9.02e-05s 2.31e-04s 2.23e-04s 2.23e-04s errno 14 pid 69923
nekrs: /__SMPI_build_dir_______________________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: size_t PAMI::Device::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed.
[lassen329:69924] *** Process received signal ***
errno 14 pid 69924
nekrs: /__SMPI_build_dir_______________________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/shmem/shaddr/CMAShaddr.h:164: size_t PAMI::Device::Shmem::CMAShaddr::read_impl(PAMI::Memregion*, size_t, PAMI::Memregion*, size_t, size_t, bool*): Assertion `cbytes > 0' failed.
[lassen329:69923] *** Process received signal ***

when running in parallel, on even 2 GPUs (single or two nodes). The modules I have loaded are

gcc/8.3.1 spectrum-mpi/2020.08.19 cuda/11.4.1

Do you have any idea what might be off? Any help would be much appreciated.

[yslan]

Is your mpi GPU-enabled?

Usually, that location is about to setup the gs-communication.
And you will get the error if you don't have GPU-awared MPI.

You can, however, turn off the GPU-awared MPI at run time by setting NEKRS_GPU_MPI=0
In that case, you will get worse scaling.

[mulligatawny]

No I don't specify a device ID anywhere...it's very strange because I've tested up to 16 nodes and 32 GPUs successfully, but can not increase to 4 GPUs per node even on one node.

[stgeke]

Try to use:

lrun -T 4 -M " -gpu" -c 1 -g 1 nekrs --setup turbPipe.par --device-id 0

btw: The problem size of turbPipe is quite small. Using more than 2-4 GPUs is probably not paying off.

Note, jsrun (lrun is just a wrapper) will map each task (MPI rank) to a resource set (using 1 CPU core and 1 GPU). So each task will only "see" one GPU with id=0 (unless you specify gpubind=off).

Why do you disable gdr (GPUDirect RDMA)? This will lower the performance.

[mulligatawny]

Thanks. This command works when I use disable_gdr. However, as given above, I see this error after the generating t-mesh step:

timing gs modes: 1.58e-04s 6.16e-04s 4.19e-04s 4.14e-04s nekrs: /__SMPI_build_dir_______________________________________/ibmsrc/pami/ibm-pami/buildtools/pami_build_port/../pami/components/devices/ibvdevice/AddressCache.h:756: bool PAMI::GPU::AddressCache::getIPCHandles(const void*, size_t, PAMI::Memregion*): As sertion cacheNode != __null && cacheNode->cudaAddr <= static_cast(reinterpret_cast<intptr_t>(buf))' failed.`

How significant do you expect the impact in performance will be when switching from RDMA to CUDA-aware?

[stgeke]

I suggest to contact your local support team. Using gdr is important for performance.

[mulligatawny]

Updating this post with the solution for reference: the support team suggested I change to an older version of cuda (<= 11.1) which did the trick. Thanks @stgeke and @yslan for all your help.