SEGFAULT with met/12.0.1 binaries on nautilus and atlantis for spack-stack-v2-rc1

[areinecke]

Description

Using spack-stack v2 rc1 metplus routines segfault under the following configurations :

• Nautilus with gcc-13.3.1: grid_stat and ascii2nc
• Nautilus with oneapi-2025.3.0: grid_stat
• Atlantis with gcc-13.4.0 and oneapi-2025.3.0: grid_stat and ascii2nc

To Reproduce

On Nautilus with gcc-13.3.1

export MODULEPATH="/p/app/projects/NEPTUNE/spack-stack/spack-stack-v2-rc1/envs/ne-gcc-13.3.1/modules/Core:$MODULEPATH"
module load stack-gcc/13.3.1
module load stack-openmpi/5.0.8
module load neptune-env/1.5.0
module load metplus
grid_stat --help
ascii2nc --help

On Nautilus with oneapi-2025.3.0

export MODULEPATH="/p/app/projects/NEPTUNE/spack-stack/spack-stack-v2-rc1/envs/ne-oneapi-2025.3.0/modules/Core:/p/app/projects/NEPTUNE/spack-stack/oneapi-2025.3.0/modulefiles:$MODULEPATH"
module load stack-intel-oneapi-compilers/2025.3.0
module load stack-intel-oneapi-mpi/2021.17
module load neptune-env/1.5.0
module load metplus
grid_stat --help
ascii2nc --help

On Atlantis with gcc-13.4.0

export MODULEPATH="/gpfs/neptune/spack-stack/openmpi-4.1.8/gcc-13.4.0/modulefiles:/gpfs/neptune/spack-stack/gcc-13.4.0/modulefiles:/gpfs/neptune/spack-stack/spack-stack-v2-rc1/envs/ne-gcc-13.4.0/modules/Core:$MODULEPATH"
module load stack-gcc/13.4.0
module load stack-openmpi/4.1.8
module load neptune-env/1.5.0
module load metplus
grid_stat --help
ascii2nc --help

On Atlantis with oneapi-2025.3.0

export MODULEPATH="/gpfs/neptune/spack-stack/spack-stack-v2-rc1/envs/ne-oneapi-2025.3.0/modules/Core:/gpfs/neptune/spack-stack/oneapi-2025.3.0/modulefiles:$MODULEPATH"
module load stack-intel-oneapi-compilers/2025.3.0
module load stack-intel-oneapi-mpi/2021.17
module load neptune-env/1.5.0
module load metplus
grid_stat --help
ascii2nc --help

Expected behavior

Expected behavior from blueback with gcc-13.3.0

grid_stat --help
DEBUG 1: Start grid_stat by reinecke(9910) at 2025-12-09 21:26:32Z  cmd: grid_stat --help
DEBUG 2: OMP_NUM_THREADS is not set. Defaulting to 1 thread. Recommend setting OMP_NUM_THREADS for faster runtimes.
DEBUG 2: OpenMP running on 1 thread(s).

*** Model Evaluation Tools (METV12.0.1) ***

Usage: grid_stat
        fcst_file
        obs_file
        config_file
        [-outdir path]
        [-log file]
        [-v level]
        [-compress level]
...

ascii2nc --help
DEBUG 1: Start ascii2nc by reinecke(9910) at 2025-12-09 21:27:06Z  cmd: ascii2nc --help

Usage: ascii2nc
        ascii_file1 [ascii_file2 ... ascii_filen]
        netcdf_file
        [-format ASCII_format]
        [-config file]
        [-mask_grid string]
        [-mask_poly file]
        [-mask_sid file|list]
        [-log file]
        [-v level]
        [-valid_beg time]
        [-valid_end time]
        [-compress level]
...

System, compiler, code, ...

See description above. Additionally, this behavior was observed prior to the update to v2-rc1.

Additional context

• May be related to MET v12.0.1 under spack-stack 1.9.3 on Hera (possibly elsewhere) raises segmentation faults #1781

[climbfuji]

I took a quick look. The successes/failures are quite random for previous test installations of the spack-stack-v2 code.

On Atlantis, I was able to get working versions of met for spack-stack-v2-rc1 by updating to met@12.1.1 (and metplus 6.1.0). This worked for GNU and OneAPI.

On Nautilus, this update did not help with GNU :-( but it helped for oneapi 2025.3.0.

I think that there is a bigger issue that needs to be resolved. And given that this feature was always broken (except for the odd lucky builds during the spack-stack-v2 development), maybe we can postpone a proper fix to the spack-stack-v2.1.0 release in February 2026? I should be able to at least get one working version on each platform, even if manually (e.g. for oneapi-2025.3.0).

[climbfuji]

I tracked this down to adding the Python library path to the rpath for met when the Python embeddings are enabled (+python).

The first question is therefore if we/any of the partners need the Python embeddings functionality in Met.

The second question then is how to fix this so that we can enable the Python bindings in case we can temporarily turn them off.