C++ & Python API for Scientific I/O with openPMD

This library provides a high-level API for writing and reading scientific data according to openPMD. It allows one to build logical file structures which drive scientific I/O libraries such as HDF5 and ADIOS through a common, intuitive interface. Where supported, openPMD-api implements both serial and MPI parallel I/O capabilities.

Usage

C++

#include <openPMD/openPMD.hpp>
#include <iostream>

// ...

auto s = openPMD::Series("samples/git-sample/data%T.h5", openPMD::AccessType::READ_ONLY);

for( auto const& i : s.iterations ) {
    std::cout << "Iteration: " << i.first << "\n";

    for( auto const& m : i.second.meshes ) {
        std::cout << "  Mesh '" << m.first << "' attributes:\n";
        for( auto const& val : m.second.attributes() )
            std::cout << "    " << val << '\n';
    }

    for( auto const& p : i.second.particles ) {
        std::cout << "  Particle species '" << p.first << "' attributes:\n";
        for( auto const& val : p.second.attributes() )
            std::cout << "    " << val << '\n';
    }
}

Python

import openpmd_api

# ...

series = openpmd_api.Series("samples/git-sample/data%T.h5", openPMD.Access_Type.read_only)

for k_i, i in series.iterations.items():
    print("Iteration: {0}".format(k_i))

    for k_m, m in i.meshes.items():
        print("  Mesh '{0}' attributes:".format(k_m))
        for a in m.attributes:
            print("    {0}".format(a))

    for k_p, p in i.particles.items():
        print("  Particle species '{0}' attributes:".format(k_p))
        for a in p.attributes:
            print("    {0}".format(a))

More!

Curious? Our manual shows full read & write examples, both serial and MPI-parallel!

Dependencies

Required:

CMake 3.11.0+
C++11 capable compiler, e.g. g++ 4.8+, clang 3.9+, VS 2015+

Shipped internally in share/openPMD/thirdParty/:

MPark.Variant 1.4.0+ (BSL-1.0)
Catch2 2.6.1+ (BSL-1.0)
pybind11 2.3.0+ (new BSD)
NLohmann-JSON 3.7.0+ (MIT)

I/O backends:

JSON
HDF5 1.8.13+ (optional)
ADIOS1 1.13.1+ (optional)
ADIOS2 2.4.0+ (optional)

while those can be built either with or without:

MPI 2.1+, e.g. OpenMPI 1.6.5+ or MPICH2

Optional language bindings:

Python:
- Python 3.5 - 3.8
- pybind 2.3.0+
- numpy 1.15+
- mpi4py 2.1+

Installation

Choose one of the install methods below to get started:

Spack

# optional:               +python +adios1 +adios2 -hdf5 -mpi
spack install openpmd-api
spack load -r openpmd-api

Conda

# optional:            OpenMPI support  =*=mpi_openmpi*
# optional:              MPICH support  =*=mpi_mpich*
conda install -c conda-forge openpmd-api

PyPI

Behind the scenes, this install method compiles from source against the found installations of HDF5, ADIOS1, ADIOS2, and/or MPI (in system paths, from other package managers, or loaded via a module system, ...). The current status for this install method is experimental. Please feel free to report how this works for you.

# we need pip 19 or newer
# optional:                   --user
python3 -m pip install -U pip

# optional:                        --user
python3 -m pip install openpmd-api

or with MPI support:

# optional:                   --user
python3 -m pip install -U pip

# optional:                                                                   --user
openPMD_USE_MPI=ON python3 -m pip install openpmd-api --no-binary openpmd-api

From Source

openPMD can then be installed using CMake:

git clone https://github.com/openPMD/openPMD-api.git

mkdir openPMD-api-build
cd openPMD-api-build

# optional: for full tests, with unzip
../openPMD-api/.travis/download_samples.sh

# for own install prefix append:
#   -DCMAKE_INSTALL_PREFIX=$HOME/somepath
# for options append:
#   -DopenPMD_USE_...=...
# e.g. for python support add:
#   -DopenPMD_USE_PYTHON=ON -DPYTHON_EXECUTABLE=$(which python)
cmake ../openPMD-api

cmake --build .

# optional
ctest

# sudo might be required for system paths
cmake --build . --target install

The following options can be added to the cmake call to control features. CMake controls options with prefixed -D, e.g. -DopenPMD_USE_MPI=OFF:

CMake Option	Values	Description
`openPMD_USE_MPI`	AUTO/ON/OFF	Parallel, Multi-Node I/O for clusters
`openPMD_USE_HDF5`	AUTO/ON/OFF	HDF5 backend (`.h5` files)
`openPMD_USE_ADIOS1`	AUTO/ON/OFF	ADIOS1 backend (`.bp` files up to version BP3)
`openPMD_USE_ADIOS2`	AUTO/ON/OFF	ADIOS2 backend (`.bp` files in BP3, BP4 or higher)
`openPMD_USE_PYTHON`	AUTO/ON/OFF	Enable Python bindings
`openPMD_USE_INVASIVE_TESTS`	ON/OFF	Enable unit tests that modify source code ¹
`openPMD_USE_VERIFY`	ON/OFF	Enable internal VERIFY (assert) macro independent of build type ²
`PYTHON_EXECUTABLE`	(first found)	Path to Python executable

¹ e.g. changes C++ visibility keywords, breaks MSVC ² this includes most pre-/post-condition checks, disabling without specific cause is highly discouraged

Additionally, the following libraries are shipped internally. The following options allow to switch to external installs:

CMake Option	Values	Library	Version
`openPMD_USE_INTERNAL_VARIANT`	ON/OFF	MPark.Variant	1.4.0+
`openPMD_USE_INTERNAL_CATCH`	ON/OFF	Catch2	2.6.1+
`openPMD_USE_INTERNAL_PYBIND11`	ON/OFF	pybind11	2.3.0+
`openPMD_USE_INTERNAL_JSON`	ON/OFF	NLohmann-JSON	3.7.0+

By default, this will build as a shared library (libopenPMD.[so|dylib|dll]) and installs also its headers. In order to build a static library, append -DBUILD_SHARED_LIBS=OFF to the cmake command. You can only build a static or a shared library at a time.

By default, the Release version is built. In order to build with debug symbols, pass -DCMAKE_BUILD_TYPE=Debug to your cmake command.

By default, tests and examples are built. In order to skip building those, pass -DBUILD_TESTING=OFF or -DBUILD_EXAMPLES to your cmake command.

Linking to your project

The install will contain header files and libraries in the path set with -DCMAKE_INSTALL_PREFIX.

CMake

If your project is using CMake for its build, one can conveniently use our provided openPMDConfig.cmake package which is installed alongside the library.

First set the following environment hint if openPMD-api was not installed in a system path:

# optional: only needed if installed outside of system paths
export CMAKE_PREFIX_PATH=$HOME/somepath:$CMAKE_PREFIX_PATH

Use the following lines in your project's CMakeLists.txt:

# supports:                       COMPONENTS MPI NOMPI HDF5 ADIOS1 ADIOS2
find_package(openPMD 0.9.0 CONFIG)

if(openPMD_FOUND)
    target_link_libraries(YourTarget PRIVATE openPMD::openPMD)
endif()

Alternatively, add the openPMD-api repository source directly to your project and use it via:

add_subdirectory("path/to/source/of/openPMD-api")

target_link_libraries(YourTarget PRIVATE openPMD::openPMD)

Manually

If your (Linux/OSX) project is build by calling the compiler directly or uses a manually written Makefile, consider using our openPMD.pc helper file for pkg-config which are installed alongside the library.

First set the following environment hint if openPMD-api was not installed in a system path:

# optional: only needed if installed outside of system paths
export PKG_CONFIG_PATH=$HOME/somepath/lib/pkgconfig:$PKG_CONFIG_PATH

Additional linker and compiler flags for your project are available via:

# switch to check if openPMD-api was build as static library
# (via BUILD_SHARED_LIBS=OFF) or as shared library (default)
if [ "$(pkg-config --variable=static openPMD)" == "true" ]
then
    pkg-config --libs --static openPMD
    # -L/usr/local/lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lopenPMD -pthread /usr/lib/libmpi.so -pthread /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so /usr/lib/libmpi.so /usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so /usr/lib/x86_64-linux-gnu/libsz.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/x86_64-linux-gnu/libdl.so /usr/lib/x86_64-linux-gnu/libm.so -pthread /usr/lib/libmpi.so -pthread /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so /usr/lib/libmpi.so
else
    pkg-config --libs openPMD
    # -L${HOME}/somepath/lib -lopenPMD
fi

pkg-config --cflags openPMD
# -I${HOME}/somepath/include

Author Contributions

openPMD-api is developed by many people. It was initially started by the Computational Radiation Physics Group at HZDR as successor to libSplash, generalizing the successful HDF5 & ADIOS1 implementations in PIConGPU. The following people and institutions contributed to openPMD-api:

Axel Huebl (HZDR, now LBNL): project lead, releases, documentation, automated CI/CD, Python bindings, installation & packaging, prior reference implementations
Fabian Koller (HZDR): initial library design and implementation with HDF5 & ADIOS1 backend
Franz Poeschel (HZDR): added JSON & ADIOS2 backend, data staging/streaming
Carsten Fortmann-Grote (EU XFEL GmbH, now MPI-EvolBio): inital Python unit tests

Transitive Contributions

openPMD-api stands on the shoulders of giants and we are grateful for the following projects included as direct dependencies:

ADIOS1 and ADIOS2 by PI S. Klasky (ORNL), team, collaborators and contributors
Catch2 by Phil Nash, Martin Hořeňovský and contributors
HDF5 by the HDF group and community
json by Niels Lohmann and contributors
pybind11 by Wenzel Jakob (EPFL) and contributors
all contributors to the evolution of modern C++ and early library preview developers, e.g. Michael Park (Facebook)
the CMake build system and contributors
packaging support by the conda-forge, PyPI and Spack communities, among others
the openPMD-standard by Axel Huebl (HZDR, now LBNL) and contributors

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

C++ & Python API for Scientific I/O with openPMD

Usage

C++

Python

More!

Dependencies

Installation

Spack

Conda

PyPI

From Source

Linking to your project

CMake

Manually

Author Contributions

Transitive Contributions

Files

README.md

Latest commit

History

README.md

File metadata and controls

C++ & Python API for Scientific I/O with openPMD

Usage

C++

Python

More!

Dependencies

Installation

Spack

Conda

PyPI

From Source

Linking to your project

CMake

Manually

Author Contributions

Transitive Contributions