Skip to content

Quick User's Guide

Jump to bottom Edit New page
Giovanni Lapenta edited this page Aug 7, 2021 · 46 revisions

Prerequisites

To work with iPic3D please be sure to have a clean Linux account to start with. The following is a list of recommendations that will make any software easy to install and manage:

  • Check your environment variables: in your home directory check the results from the command env. Verify that your environment is clean, in particular the variables $PATH and $LD_LIBRARY_PATH should not point directories different than the standard Linux paths (/usr/local, /usr/lib, etc).

  • Use modules to manage all your libraries: for all the libraries like PETSc, HDF5, H5hut, and for tools like CMake, valgrind, etc., you should make use of the Linux Modules. This is the best way to maintain multiple software in your system, including multiple versions. Consider also installing all the libraries and software in a path like /usr/local/ or /opt/ owned by root, but accessible to all users.

  • Do not install libraries in the system directory: many Linux distributions, like Debian, CentOS and Ubuntu, allow you to install libraries, e.g. HDF5, from the command line using a simple package manager. For example, in Ubuntu you can type apt install hdf5 to install HDF5. We strongly encourage you to use these method only for your system packages, and not for the software and libraries that you're going to use for programming. The libraries installed with such method will be pre-compiled with a different set of options than the ones that you need, and they will be place in the system path /usr/lib (or similar), which will interfere with the automatic configuration of the code using CMake.

To work with the current version of iPic3D you need to install the following libraries and tools:

  • MPI: Any current version of the MPI parallel library will be fine.
  • HDF5: version 1.8.11 or previous. This library is used for the I/O in iPic3D. It needs to be compiled using the --enable-parallel flag for parallel I/O. A detailed walk through is be presented bellow.
  • H5hut: This library is an interface between iPic3D and parallel HDF5, mainly for the handling of particle input and output. The source code is included in the sources. A walk through its installation is given bellow.
  • CMake: version 3 or higher. This is used by iPic3D for the automatic detection of the library paths, and generation of the Makefiles.
  • git: is a software management program that allows you to download, keep up to date, and track your modifications of the code.

Downloading iPic3D

Using git:

mkdir ~/Sources
cd ~/Sources
git clone https://github.com/CmPA/iPic3D iPic3D-git

where ~Sources is the path where you want to install iPic3D, and iPic3d-git is the name that you want to give to the iPic3D directory.

Compiling the libraries

Purging your system

The biggest issue installing iPic3D (and in general any other software) in Linux is having an untidy Linux environment. Linux is based on identify everything as a file. Your Linux environment tells the computer where those files are located:

  • So, for example, if you run the command ls the system looks into the directories listed in $PATH trying to find that command. The first time it finds it, it will run that particular command. So the order of the directories in the $PATH variable is important.

  • The second most important environment variable is $LD_LIBRARY_PATH, which is a list of the directories in which the system will look for a given software library. The order is again very important.

  • In addition to these two, additional variables can have an impact on the functioning of your system, and they depend on particular software. For example, the variable PYTHONPATH indicates to Python where to look for packages.

These variables are set using two methods: with the modules package, or with your interpreter configuration file geenrally called .bashrc (some systems use other interpreters than bash, for example csh, tcsh, ksh, etc. Small differences can be observed among interpeters).

We recommend to use modules as much as possible.

Otherwhise, make sure that your .bashrc is clean and that all the variables are pointing to the directories that you intent to point to.

If you make any modifications to your .bashrc file, the recommended procedure to re-read this file is:

cd ~/
export PATH=""
export PATH=/bin:/usr/bin
source ~/.bashrc
env

In csh and tcsh this should read instead:

cd ~/
setenv PATH /bin:/usr/bin
setenv PATH ""
source ~/.cshrc
env

The last command, env, should give you an overview of all the variables defined in your environment. Make sure they point to the correct addresses.

Compiling Linux modules

Download the sources from the this webpage.

cd ~/Sources
mv ~/Downloads/modules-3.2.10.tar.gz .
tar xvf modules-3.2.10.tar.gz
cd modules-3.2.10
./configure
make
sudo make install

If you are installing this software in a Mac OS X, make sure to install de XCode. The TCL library is hidden deep in this kit:

cd ~/Sources/modules-3.2.10
./configure --with-tcl-lib=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/Tcl.framework/ --with-tcl-inc=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/Tcl.framework/Headers
make
sudo make install

Compiling MPICH

Download the sources:

cd ~/Sources
wget http://www.mpich.org/static/downloads/3.2/mpich-3.2.tar.gz
tar xvf mpich-3.2.tar.gz
cd mpich-3.2

Remember that in Mac OS X the downloading command, wget, does not exist. You can use:

curl "http://www.mpich.org/static/downloads/3.2/mpich-3.2.tar.gz" -o "mpich-3.2.tar.gz"

Or you can just download the sources using your web browser.

Configure and compile MPICH with the following options:

./configure --prefix=/usr/local/mpich/3.2 --enable-shared --with-pm=hydra --disable-fortran
make
sudo make install

Compiling HDF5 1.8.11 (Use 1.10.7 for newer iPic3D versions e.g. Coils branch)

For compatibility reasons with the library H5hut, we are forced to use the version 1.8.11 of HDF5 (we are planning to get rid of H5hut in the near future!). Many systems do not propose this library, but it can be installed very easily.

ATTENTION: For new code versions and in particular for the branch Coils, the newer library version 1.10.6 is needed.

Download HDF5 version 1.8.11 to your home system:

cd ~/Sources
wget https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.8.11/src/hdf5-1.8.11.tar.bz2
tar xvf hdf5-1.8.11.tar.bz2
cd hdf5-1.8.11

Make sure to load the MPI module, it will be needed to compile the parallel options:

module load mpich

where mpi is the correct module corresponding to your MPI library.

Configure the source code to use the correct options:

CC=mpicc CXX=mpicxx ./configure --prefix=/usr/local/hdf5/1.8.11 --enable-parallel --enable-hl --enable-shared --enable-static
make
sudo make install

where /usr/local/hdf5/1.8.11 is the installation path. This particular path requires root access. If you do not have root access you can still point the library to a path in your home. Make sure that the lib and include directories have been correctly installed in this path.

If the previous command gives the error message CC=mpicc unknown command, it is very possible that you're using a csh or tcsh shell. Use instead the following command (include the parenthesis!):

(setenv CC mpicc; setenv CXX mpicxx; ./configure --prefix=/usr/local/hdf5/1.8.11 --enable-parallel --enable-hl --enable-shared --enable-static)
make
sudo make install

Remember that if you are using Intel compilers and Intel MPI (IMPI), the compiler is CC=mpiicc and CXX=mpiicpc.

Mac OS X Warning: for some versions of the Xcode C compiler (clang) this configuration/compilation may fail because of an outdated compiler. Check carefully that the static and dynamic libraries have been correctly compiled. You might want to try the following option in the compilation line, after the CC and CXX options: CFLAGS=“-std=c99”.

Create a new Linux Module for your installation of HDF5 as shown in the section bellow.

Compiling H5hut

Load the HDF5 module and the MPI library:

module load hdf5 mpi

where hdf5 and mpi are the names of the module files in your system.

If you are not using modules, please remember to keep you system paths clean and remember to add the following lines to your .bashrc file:

export PATH=/usr/local/hdf5/1.8.11:/usr/local/hdf5/1.8.11/bin:/usr/local/hdf5/1.8.11/incude:$PATH
export LD_LIBRARY_PATH=/usr/local/hdf5/1.8.11/lib:/usr/local/hdf5/1.8.11/lib64:$LD_LIBRARY_PATH

The source code of H5hut is included in the iPic3D external directory. Extract the file:

cd ~/Sources
cp ~/Sources/iPic3D-git/external/H5hut-1.99.12.tar.bz2 .
tar xvf H5hut-1.99.12.tar.bz2
cd H5hut-1.99.12

ATTENTION: For new code versions and in particular for the branch Coils, the newer library version 2.0.0rc3 is needed. It can be obtained from: https://gitlab.psi.ch/H5hut/src/-/wikis/Installation/Download

Configure and compile the library. We have observed four possible scenarios for this compilation:

  • Case 1: all the libraries are correctly installed and the modules work flawlessly:
./configure --prefix=/usr/local/H5hut/1.99.12 --enable-parallel --enable-large-indices --enable-shared --enable-static
make
sudo make install
  • Case 2: the hdf5 files detected by the configuration do not point to the right path. You can check this visually inspecting the last lines of the output after the configuration line, or the config.log file. You can force the configuration to look for the correct path using:
./configure --prefix=/usr/local/H5hut/1.99.12 --enable-parallel --enable-large-indices --with-hdf5=/usr/local/hdf5/1.8.11 --enable-shared --enable-static
  • Case 3: the hdf5 files detected are wrong and the configuration does not detect the correct mpi library or gives and error related to the mpi compiler. You can force the configuration to use the correct mpi compilers:
CC=mpicc CXX=mpicxx ./configure --prefix=/usr/local/H5hut/1.99.12 --enable-parallel --enable-large-indices --with-hdf5=/usr/local/hdf5/1.8.11 --with-mpi=/path/to/mpi --enable-shared --enable-static
  • Case 4: you tried case 3 but it gives you an error message saying that CC=mpicc unknown command. In this case it is very possible that you're using a csh or tcsh shell. Use instead the following command (include the parenthesis!):
(setenv CC mpicc; setenv CXX mpicxx; ./configure --prefix=/usr/local/H5hut/1.99.12 --enable-parallel --enable-large-indices --with-hdf5=/usr/local/hdf5/1.8.11 --with-mpi=/path/to/mpi --enable-shared --enable-static)

One of the mos common errors that we have observed shows up during the compilation, where some of the lines read:

/home/myuser/H5hut-1.99.12/src/h5core/h5_hdf5_private.h:952: undefined reference to `H5Pset_dxpl_mpio'
/home/myuser/H5hut-1.99.12/src/h5core/h5_hdf5_private.h:937: undefined reference to `H5Pset_fapl_mpiposix'

This error comes because you're trying to link H5hut to a serial version of HDF5. This can happen for multiple reasons:

  1. you have a version of HDF5 installed in the system directories (/usr/lib:/usr/lib64). Solution: you can uninstall the system version, or you can pre-pend the HDF5 directory to the system paths, so it is picked up first before the system libraries (export PATH=/usr/local/hdf5/1.8.11:/usr/local/hdf5/1.8.11/bin:/usr/local/hdf5/1.8.11/incude:$PATH, and export LD_LIBRARY_PATH=/usr/local/hdf5/1.8.11/lib:/usr/local/hdf5/1.8.11/lib64:$LD_LIBRARY_PATH) ,

  2. you gave the wrong directory to the --with-hdf5 flag. Solution: relaunch the configuration using the correct path,

  3. you are pointing to the correct HDF5 directory but, for some reason the HDF5 compilation created the lib64 directory instead of lib. Solution: if the solution 1 does not help, you can try forcing the compiler to detect the correct library path, by using the following configuration line:

LDFLAGS=-L/usr/local/hdf5/1.8.11/lib64 CC=mpicc CXX=mpicxx ./configure --prefix=/usr/local/H5hut/1.99.12 --enable-parallel --enable-large-indices --with-hdf5=/usr/local/hdf5/1.8.11 --with-mpi=/path/to/mpi --enable-shared --enable-static

All the paths mentioned above must point to the correct paths in your system.

Compiling PETSc

Download and expand the source code from the PETSc website

cd ~/Sources
wget http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-lite-3.7.6.tar.gz
tar xvf petsc-lite-3.7.6.tar.gz
cd petsc-lite-3.7.6

The following commands will configure the code. This procedure may take a very long time, and if it fails at any given point, you need to re-make the complete configuration again. So at this point you must be patient and maybe grab a paper to read.

./configure --prefix=/usr/local/petsc/3.6.0-mpich --with-mpi-dir=/usr/local/mpich/3.2 --download-f2cblaslapack=1 --with-fc=0 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-x=0 --with-debugging=0

This command downloads and compiles the BLAS and LAPACK libraries. For some weird reason this process fails in some systems, saying that you don't have access to the directory given in the --prefix flag. This happens when you're trying to install the library in a system directory. To avoid the issue, before configuring the library create the following directories:

mkdir -p /usr/local/petsc/3.6.0-mpich/lib
mkdir -p /usr/local/petsc/3.6.0-mpich/include
mkdir -p /usr/local/petsc/3.6.0-mpich/share
mkdir -p /usr/local/petsc/3.6.0-mpich/bin

BLAS and LAPACK are in general available in most of the systems, so you might want to use them instead. In such case you must drop the --download-f2cblaslapack=1 flag and use instead --with-blas=/path/to/blas/library --with-lapack=/path/to/lapack/library.

Creating the Linux modules for the compiled libraries

If you do not have root access you can create your own Linux Modules in your home directory.

Create a directory:

cd ~
mkdir mymodules
mkdir mymodules/hdf5
mkdir mymodules/H5hut

Add this directory to the list of available modules by inserting this line in your .bashrc:

echo 'module use ~/mymodules' >> ~/.bashrc

Create a file called ~/mymodules/hdf5/1.8.11 to contain the following information for the hdf5 module:

#%Module1.0#####################################################################
##
## modules hdf5/1.8.11
##
## hdf5/1.8.11  Written by Jorge Amaya
##
proc ModulesHelp { } {
        global version modroot

        puts stderr "hdf5/1.8.11 - sets the Environment for the HDF5 1.8.11 PARALLEL library"
}

module-whatis   "Sets the environment for the HDF5 1.8.11 parallel library"

# for Tcl script use only
set     topdir          /usr/local/hdf5/1.8.11
set     version         1.8.11
set     sys             x86_64

setenv          HDF5_DIR            $topdir
setenv          HDF5_ROOT           $topdir
setenv          HDF5_HOME           $topdir
prepend-path    PATH                $topdir/include
prepend-path    PATH                $topdir/bin
prepend-path    LD_LIBRARY_PATH     $topdir/lib

where we have included the definition of three environment variables, and we have added paths at the front of the $PATH and the $LD_LIBRARY_PATH variables. You can test new by loading the new moudle:

module load hdf5/1.8.11
echo $PATH
echo $LD_LIBRARY_PATH
echo $HDF5_HOME

You can now copy this file and do the same for the H5hut library.

Compiling iPic3D

Load the libraries:

module load HDF5 mpi H5hut

Create a compiling directory:

cd ~/Sources/iPic3D-git
mkdir build
cd build

Launch the automatic generation of the Makefile:

cmake ..

Enter the GUI to select particular options:

ccmake ..

In the GUI turn ON/OFF options moving up and down with the arrow keys, and switching options with the ENTER key. For example, turn ON the option IPIC_H5HUT_OUTPUT to activate the parallel HDF5 options.

Once all the important options are switched ON, press the key c to configure the Makefile. This might list on the top of the screen a new set of variables that need to be setup (i.e. paths). Be sure that the options are correctly filled. Type once again the key c and fill the additional options, until the key g (generate) is available. This last key will create the Makefile! Exit the GUI pressing the key e.

If there are no errors, you can compile the code now:

make clean
make

This will create the file iPic3D in the current directory.

Common errors and issues during compilation

Whenever you encounter a compilation error, it is a good idea to recompile the code using the following command:

make clean
make VERBOSE=1

This will show the full compilation lines and can point towards an error in the setting of a path or a missing flag.

These are the most commonly encountered errors and issues that we have observed during the compilation of the code:

  • Error: Library not compatible, or library path is wrong.

    Diagnostic: CMake is not finding the correct library path, it is finding a wrong version or it is finding the system default libraries installed in /usr/lib.

    Solution 1: Check that you loaded the correct module. Check that the module is correctly configured, using the command module show.

    Solution 2: Give the good path to CMake. The first path it searches, even before the system paths, is $CMAKE_PREFIX_PATH, so include the option in the CMake call using the command:

cmake -DCMAKE_PREFIX_PATH=/my/good/path ..

  • Error: SEEK_SET error when using Intel compilers and Intel MPI.

    Diagnostic: There is an incompatibility of definitions when using MPI with the Intel Compiler. This error has been documented in the Intel website.

    Solution: include the flags and in the compilation options, wither in the ccmake GUI or using the command line:

cmake -DCMAKE_CXX_FLAGS='-DMPICH_IGNORE_CXX_SEEK -DMPICH_SKIP_MPICXX' ..

  • Error: the sources compile, but the linking falils.

    Diagnostic: there is a library incompatibility or missing path in the system.

    Solution: check that you're pointing to the correct paths, and check that your environment variables, including $LD_LIBRARY_PATH, are pointing to the correct paths in the system

Compilation on SuperMUC

TBD

Libraries using brew on MAC

  • Install Xcode form Apple Store

  • Install brew:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
  • Install cmake, openmpi and petsc:
brew install cmake

brew install openmpi

brew install petsc 
  • Unfortunately for compatibility with H5HUT you need to install your own version of hdf5 1.10.7 following hte instructions above. The brew provided version of hdf5 or hdf5-mpi is not compatible with H5HUT.
Add a custom footer
Add a custom sidebar
Clone this wiki locally