libtrixi: Serving legacy Fortran codes with simulations in Julia

This is the companion repository for the talk

Serving legacy Fortran codes with simulations in Julia Gregor Gassner, Benedict Geihe, Michael Schlottke-Lakemper

Juliacon2024, Eindhoven, the Netherlands, 9th to 13th July 2024

Abstract

With libtrixi we present a blueprint for connecting established research codes to modern software packages written in Julia without sacrificing performance. Specifically, libtrixi provides an API to Trixi.jl, a Julia package for adaptive numerical simulations of conservation laws.

Slides

The slides can be downloaded here.

Reproducibility

The instructions are written based on the supercomputer JUWELS, operated at Jülich Supercomputing Centre, on which the results presented on the poster were obtained.

Getting started

The following standard software packages need to be made available locally before installing libtrixi:

• C compiler with support for C11 or later (e.g., GCC or Clang) (we used GCC v12.3.0)
• Fortran compiler with support for Fortran 2018 or later (e.g., Gfortran) (we used Gfortran v12.3.0)
• CMake (we used cmake v3.26.3)
• MPI (e.g., Open MPI or MPICH) (we used Open MPI v4.1.5)
• HDF5 (we used parallel HDF5 v1.14.2)
• Julia (it is advised to use tarballs from the official website or the juliaup manager, we used v1.10.3)
• Paraview for visualization (we used v5.12.0)

The following software packages require manual compilation and installation. To simplify the installation instructions, we assume that the environment variable WORKDIR was set to the directory, where all the code is downloaded to and compiled, and PREFIX was set to an installation prefix. For example,

export WORKDIR=$HOME/repro-poster-2024-pasc-libtrixi
export PREFIX=$WORKDIR/install

t8code

t8code is a meshing library which is used in Trixi.jl. The source code can be obtained from the official website. Here is an example for compiling and installing it (we used v2.0.0):

cd $WORKDIR
git clone --branch 'v2.0.0' --depth 1 https://github.com/DLR-AMR/t8code.git
cd t8code
git submodule init
git submodule update
./bootstrap
./configure --prefix="${PREFIX}/t8code" \
            CFLAGS="-Wall -O2" \
            CXXFLAGS="-Wall -O2" \
            --enable-mpi \
            CXX=mpicxx \
            CC=mpicc \
            --enable-static \
            --without-blas \
            --without-lapack
make
make install

libtrixi

Here is an example for compiling and installing libtrixi (we used v0.1.5):

cd $WORKDIR
git clone --branch 'v0.1.5' --depth 1 https://github.com/trixi-framework/libtrixi.git
cd libtrixi
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_INSTALL_PREFIX=${PREFIX}/libtrixi ..
make
make install

A separate Julia project for use with libtrixi has to be set up:

mkdir ${PREFIX}/libtrixi-julia
cd ${PREFIX}/libtrixi-julia
${PREFIX}/libtrixi/bin/libtrixi-init-julia \
  --t8code-library ${PREFIX}/t8code/lib/libt8.so \
  ${PREFIX}/libtrixi

To exactly reproduce the results presented on the poster, the provided Manifest.toml can be used to install the same package versions for all Julia dependencies:

cp Manifest.toml ${PREFIX}/libtrixi-julia/
cd ${PREFIX}/libtrixi-julia
JULIA_DEPOT_PATH=./julia-depot \
  julia --project=. -e 'import Pkg; Pkg.instantiate()'

Alternatively all packages can be updated to the most recent version on demand:

cd ${PREFIX}/libtrixi-julia
JULIA_DEPOT_PATH=./julia-depot \
  julia --project=. -e 'import Pkg; Pkg.update()'

Trixi2Vtk

Trixi2Vtk needs to be installed for later post processing (we used v0.3.15):

cd ${PREFIX}/libtrixi-julia
JULIA_DEPOT_PATH=./julia-depot \
  julia --project=. -e 'import Pkg; Pkg.add(name="Trixi2Vtk", version="0.3.15")'

Running the baroclinic instability simulation

The julia scripts for setting up simulations (libelixirs) can be found in the examples folder:

export EXAMPLES=$PREFIX/libtrixi/share/libtrixi/LibTrixi.jl/examples

For the results on the poster the resolution of the computational mesh was increased. For this, modify line 128 of ${EXAMPLES}/libelixir_p4est3d_euler_baroclinic_instability.jl such that initial_refinement_level = 1.

We used the following SLURM script to launch the simulation on JUWELS:

#!/bin/bash -x
#SBATCH --nodes=128
#SBATCH --ntasks-per-node=48

module purge
module load GCC
module load OpenMPI
module load HDF5/1.14.2

srun -n "${SLURM_NTASKS}" \
  ${PREFIX}/libtrixi/bin/trixi_controller_simple_f \
  ${PREFIX}/libtrixi-julia \
  ${EXAMPLES}/libelixir_p4est3d_euler_baroclinic_instability.jl

Output files will be written to ./out_baroclinic and need to be post processed:

cd ${PREFIX}/libtrixi/out_bubble
JULIA_DEPOT_PATH=${PREFIX}/libtrixi-julia/julia-depot \
   julia --project=${PREFIX}/libtrixi-julia -e 'using Trixi2Vtk; trixi2vtk("solution*.h5")'

The results can now be viewed using, e.g., paraview. We used slice_surface.pvsm to generate our results.

Authors

This repository was initiated by Benedict Geihe and Michael Schlottke-Lakemper.

License

The contents of this repository are licensed under the MIT license (see LICENSE.md).