The PI-Control experiment of E3SMv2.1 is suddenly broken at year 0438-08-04_14:00:00

[shipf16]

Hello, everyone

I meet this technical problem that the PI-Control experiment is suddenly broken in the time of 0438-08-04_14:00:00 using E3SMv2.1 on Perlmutter. I sincerely apply for your suggestions and help.

I submit new jobs to run the PI-Control experiment from the beginning time of 0391-01-01 using E3SMv2.1. This PI-Control experiment has finished 47 years. However, the PI-Control experiment is suddenly broken at the time of 0438-08-04_14:00:00 with the attached e3sm_log. I check the previous value of 0438-07 is normal. I really don’t know how to resolve this problem. Could you please provide me with further guidance and help?

Please accept my apologies for any inconvenience caused and thank you very much! I would greatly appreciate any guidance or advice you could provide to help me resolve this problem.

The below is e3sm_log for your further review:

[rljacob]

Are you using the maint-2.1 branch? First try running again from the last restart to make sure its repeatable.

[shipf16]

Hello, Thank you very much for your reply! Yes, I am using the maint-2.1 branch for E3SMv2.1 on Perlmutter. I also have tried to rerun again but all tests failed at the time 0438-08-04_14:00:00. Could you please tell me more suggestions to help me resolve this problem? Thank you very much for your kind suggestions!