MPAS is hanging when it is reading RRTMG_SW_DATA

[uturuncoglu]

When MPAS is running under NUOPC cap and external communicator passed through mpas_init(), the model is hanging when reading RRTMG_SW_DATA file. I could able to attach the gdb to hanging process and collect backtrace that can be seen in the following log file.

bt-dec2323_233164.txt

It seems that the model is hanging in src/core_atmosphere/physics/mpas_atmphys_rrtmg_swinit.F source file line 206. I also put couple of print statement around the code and it looks like it is hanging when doing broadcast with DM_BCAST_REAL. So, there could be some issue about using MPI Communicator passed from the ESMF library.

I could able to run the model after setting following options in namelist.atmosphere since it does not read those files,

&physics
...
....
    config_radt_lw_scheme = 'off'
    config_radt_sw_scheme = 'off'
/

But, these options need to be activated to do scientific runs and underlying issue needs to be fixed in the future. My version of MPAS-Model is based on following commit from develop branch,

commit 03eef4810a6e27d0c9b9d5b44e71495d9a5a8a58 (origin/develop)
Merge: 1c3c6e4c c6288363
Author: Michael Duda <duda@ucar.edu>
Date:   Mon Mar 23 15:10:09 2026 -0600

[uturuncoglu]

JFYI, @mgduda @scrasmussen @danrosen25 @anntsay

[danrosen25]

@uturuncoglu @mgduda @scrasmussen @anntsay
The external sub-communicator, defined by ESMF for each modeling component, must be passed into MPAS otherwise MPAS will try to run all MPI calls over MPI_COMM_WORLD.

In the mpas_dmpar_init subroutine it has an optional argument for the external communicator.

MPAS-Model/src/framework/mpas_dmpar.F

Line 272 in b9090a1

subroutine mpas_dmpar_init(dminfo, external_comm)!{{{

There are two steps to getting the communicator for the ESMF modeling component

1. ESMF_GridCompGet - to get the the VM from the ESMF_GridComp (model)
   https://earthsystemmodeling.org/docs/nightly/develop/ESMF_refdoc/node4.html#SECTION04034700000000000000

2. ESMF_VMGet - to get the mpi communicator from the VM
   https://earthsystemmodeling.org/docs/nightly/develop/ESMF_refdoc/node6.html#SECTION0601361800000000000000

[uturuncoglu]

@danrosen25 Thanks for the information. In my case, I have single component (MPAS) and it uses all the processor. I am also passing communicator using mpas_init. So, I think something else going on here. The model works fine after disabling read of those files. Anyway, I'll look at it more maybe I have a functional fully coupled system.