Add high resolution support for remaining tasks

[forsyth2]

#136 added high resolution support for climo and ts. The other zppy tasks still need high resolution support.

[xylar]

It might be worth considering a config option similar to campaign to distinguish between "low", "medium" and "high" resolution. (I put these in quotes because the distinction is subjective.)

MPAS-Analysis run on high-res data provided by @ndkeen required:

• PostMOC = True because high res run was conducted with an old version of the MOC online analysis that does not use the regions or dimension names required by MPAS-Analysis
• Turning off all analysis involving anomalies because the reference year isn't available:

generate = 'all', 'no_landIceCavities', 'no_BGC', 'no_icebergs', 'no_min', 'no_max', 'no_sose', 'no_climatologyMapAntarcticMelt', 'no_regionalTSDiagrams', 'no_timeSeriesAntarcticMelt', 'no_timeSeriesOceanRegions', 'no_climatologyMapSose', 'no_woceTransects', 'no_soseTransects', 'no_geojsonTransects', 'no_oceanRegionalProfiles', 'no_hovmollerOceanRegions',  'no_climatologyMapOHCAnomaly', 'no_timeSeriesOHCAnomaly', 'no_timeSeriesSSHAnomaly', 'no_timeSeriesSalinityAnomaly', 'no_timeSeriesTemperatureAnomaly', 'no_refYearMpasClimatologyOcean'

I think this may be turning off more analysis than it's leaving on so we probably want to switch to explicitly specifying the analysis to run.

• Setting parallelTaskCount = 1 and ncclimoThreads = 1 to prevent OOM errors.

@ndkeen, were there other changes that I'm forgetting?

[ndkeen]

I think those were the changes -- but still trying to complete the analysis.

[xylar]

I was able to complete 10 years (climatologies and time series) of high-res MPAS-Analysis on the large memory nodes on Cori in well under an hour. It seems like that might be the main plausible way of doing MPAS-Analysis at high res.

[xylar]

I don't know exactly when but I will try to add this support sometime in the next week or 2.

[forsyth2]

I don't know exactly when but I will try to add this support sometime in the next week or 2.

Thanks @xylar!

[xylar]

@forsyth2, I will try to make a PR for this tomorrow.

[forsyth2]

Note: resolved partially by #176

[forsyth2]

Note: test using E3SM Diags via zppy with high resolution. It may already be supported.

[darincomeau]

I'm actually trying to run zppy on a v1 HR simulation I'm running on Chrysalis now. So far unsuccessfully, but is likely user error.

[xylar]

@darincomeau is running out of memory when running 12 ocean ncclimo threads on a Chrysalis node. We may need to add some Chrysalis-specific config options to the HR setup based on his experience running standalone MPAS-Analysis. We'll see what turns up.

Questions remain about the e3sm_diags and global time series analysis. @darincomeau, do you want to post the path to your results so maybe @forsyth2 can help debug?

[darincomeau]

The run is here:
/lcrc/group/acme/ac.dcomeau/scratch/chrys/E3SMv1/v1.HR.piControl.advection-bug-fix
and the config file is under zppy, and the zppy output under zppy_output.

[forsyth2]

Questions remain about the e3sm_diags and global time series analysis.

$ cd /lcrc/group/acme/ac.dcomeau/scratch/chrys/E3SMv1/v1.HR.piControl.advection-bug-fix/zppy_output/post/scripts
$ grep -v "OK" *status
climo_atm_monthly_180x360_aave_0021-0025.status:ERROR (2)
mpas_analysis_ts_0021-0025_climo_0021-0025.status:ERROR (1)

climo is a dependency of e3sm_diags and mpas_analysis is a dependency of global_time_series, so since we have the failures listed above, the later tasks won't even run.