E3SM Error in advance_xp2_xpyp and club_tend_cam

[CF513]

I am running E3SM-maint2.0 on Perlmutter (pm-cpu).
git clone -b maint-2.0 --recursive https://github.com/E3SM-Project/E3SM.git

The case that reported errors is the cloud-feedback 10-year simulation with COMPSET=F2010-CICE.
The running scripts work well on cori, and some similar simulations on Perlmutter after some change in autoconversion, but the following error appears in the control run (no code modification) at 3yr 6mon 28d as shown in the log files, and the simulation outputs data monthly.

Error in e3sm.log:

Error in advance_xp2_xpyp
wm_zm = NaN NaN ......
wpthlp = NaN NaN ......
(and other variabless = NaN NaN .....)

ERROR: clubb_tend_cam: Fatal error in CLUBB libraryERROR in /global/u2/f/fengc/Perl_maint2.0/components/eam/src/physics/cam/clubb_intr.F90 at line 2296

Error calling advance_xp2_xpyp
Fatal error in CLUBB: at timestep 43591 LAT (radians): -1.0882411338464488 L ON (radians): 5.5781896035537422 LAT (degrees): -62.351624061932839 LON (degrees) : 319.6067216073 Global Column Number: 17074

MPICH ERROR [Rank 9] [job id 11172479.0] [Thu Jul 6 03:11:29 2023] [nid004388] - Abort(100 1) (rank 9 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 9

Warnings:

MPIIO WARNING: DVS stripe width of 24 was requested but DVS set it to 1 (showed up multiple times)

I have run 8 similar simulations (very tiny difference in auto conversion rate), but only 2 of them stopped due to the above error after 3 year 5 months' and 3 year 6 months' output. No error was found in ocn/lnd/ice/atm.log.

My e3sm.log file and run scripts are attached below:
e3sm.log.11172479.230706-000715.txt

20220511.F2010-CICE.ctl.sh.txt
20220511.F2010-CICE.4k.sh.txt

Thank you very much for your time!

[ndkeen]

I just pushed a merge to maint-2.0 to add Intel compiler for pm-cpu.
In #5810, I also describe one change that is
needed to scorpio (which is a submodule) to build.
Is it easy to try your case with Intel?
create_newcase ..... --compiler=intel ....

[CF513]

The error disappeared after the change you made. Thank you a lot!