tmLQCD QUDA

Machines

Jureca

The default (GPU) modules on Jureca don't seem to work very well, use the following set:

CUDA/8.0.44 GCCcore/.5.4.0  binutils/.2.27 GCC/5.4.0 Bison/.3.0.4 \
MVAPICH2/2.2-GDR flex/2.6.0 ncurses/.6.0  CMake/3.6.2 \
OpenBLAS/0.2.19-LAPACK-3.6.1

This allows you to compile QUDA with the MG enabled. Use /usr/local/software/jureca/Stages/2016b/software/GCCcore/5.4.0/bin/g++ (or the equivalent at time of reading) as both host and CMAKE compiler. This should allow you to build a consistently working QUDA. Other combinations, such as using ICC mixed with GCC or different versions of GCC, which is default auto-detected by CMake, will fail.

Follow these instructions to build QUDA with the modern CMake build system: https://github.com/lattice/quda/wiki/Building-QUDA-with-cmake

For tmLQCD, we still need to figure out if we can use the Intel compiler at the same time. For now, this configure line gets it built with GCC:

${path_to}/configure --enable-halfspinor --enable-gaugecopy \
--with-limedir=${path_to}/lime --enable-mpi --with-mpidimension=4 \
--enable-omp --disable-sse2 --disable-sse3 --enable-alignment=32 \
--with-lapack="-lopenblas" F77=gfortran CC=mpicc CXX=mpicxx \
--with-qudadir=${path_to}/quda/build \
--with-cudadir=/usr/local/software/jureca/Stages/2016b/software/CUDA/8.0.44 \
LDFLAGS=-lcuda

Notes

• It should be noted that we use OpenBLAS because it seems that MAGMA is easier to build that way.

Dependencies

• QUDA
  • QUDA-MG
    • MAGMA 1.7.0
  • QUDA test programs
    • if gauge I/O is required QIO / QMP

QUDA-MG depends on MAGMA 1.7.0

To use the QUDA-MG, MAGMA 1.7.0 is presently required. It is most conveniently built using the available Makefile and the set of modules noted above. g++ should be used as a compiler.

First

$ cd ${path_to_magma-1.7.0_sources}
$ ln -s make.inc.openblas make.inc

set

LIB = -lopenblas -lstdc++ -lm
prefix=${magma_install_dir} ## set this to desired installation path

in make.inc

Then:

$ export CUDADIR=$EBROOTCUDA
$ export OPENBLASDIR=$EBROOTOPENBLAS
$ make -j && make install

When these instructions have been followed, it is easy to specify QUDA_MAGMAHOME=${magma_install_dir} when configuring QUDA with the MG.

QUDA test program dependencies

To run the QUDA tests, you will require QMP and QIO configured and compiled, which is somewhat tricky because of build system inconsistencies.

QMP

This is straight-forward to compile and can be done in an external build directory, simply by doing the following configure:

$ ${path_to_qmp}/configure --prefix=${qmp_install_dir} CC=mpicc CFLAGS=-std=c99 --with-qmp-comms-type=MPI
$ make && make install

LIME & QIO

While qio can also be built in an external directory without problems, there will be issues linking against lime and finding lime.h. As a result, you should compile lime in its source directory to begin with.

## in lime directory
$ ./configure 
$ make

## in qio build directory
$ ${qio_source_dir}/configure
$ cd other_libs
$ rm c-lime
$ ln -s ${lime_source_dir} c-lime  ## this is necessary
$ cd ..
$ ${qio_source_dir}/configure --enable-largefile --with-qmp=${qmp_install_dir} CC=mpicc CFLAGS=-std=c99 --prefix=${qio_install_dir}
$ make && make install