Skip to content

Commit e0ea410

Browse files
natalie-perlinNatalie PerlinNatalie Perlingspetro-NOAA
authored
[release/public-v2.2.0] Update MacOS configuration and compatibility (#953)
Updated modulefiles, scripts and configurations for running SRW v2.2.0 on MacOS platforms, summarized below. The changes include adaptation of the code for two MacOS architecthures, Intel/x86_64 and M1/arm64 (could be aarch64 if using Rosetta) * modulefiles in ./modulefiles, ./modulefiles/tasks/macos/, srw_common.lua (successfully tested on all platforms but cloud) * machine file in ./ush/machine/macos.yaml * ./etc/lmod-setup.* * scripts in ./ush/wrappers/* ./jobs/ , ./scripts/*.sh - the header changed to #!/usr/bin/env bash; a small bug in one of the scripts * a patch file is applied for the ./sorc/CMakeLists.txt to build SRW on M1/arm64 machine; patch is located in ./patches/patch_macos_arm64_sorc_cmakelists.txt, and it is applied in ./devbuild.sh when needed * small fixes in ./ush/job_preamble.sh, ./ush/preamble.sh (use mac-specific commands for runtime estimates) * fix for ./ush/python_utils/filesys_cmds_vrfy.py: function vrfy_ln creates links with gnu-ln on MacOS (from coreutils) * updates to the documentation for the MacOS users and adding options to install workflow conda environments. --------- Co-authored-by: Natalie Perlin <[email protected]> Co-authored-by: Natalie Perlin <[email protected]> Co-authored-by: Gillian Petro <[email protected]>
1 parent 4f5573b commit e0ea410

File tree

99 files changed

+332
-193
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

99 files changed

+332
-193
lines changed

README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ https://epic.noaa.gov/wp-content/uploads/2022/12/Debugging-Guide.pdf
1515

1616
The SRW App v2.2.0 citation is as follows and should be used when presenting results based on research conducted with the App:
1717

18-
UFS Development Team. (2023, Oct. 30). Unified Forecast System (UFS) Short-Range Weather (SRW) Application (Version v2.2.0). Zenodo. https://doi.org/10.5281/zenodo.10015544
18+
UFS Development Team. (2023, Oct. 31). Unified Forecast System (UFS) Short-Range Weather (SRW) Application (Version v2.2.0). Zenodo. https://doi.org/10.5281/zenodo.10015544
1919

2020
[![Python unittests](https://github.com/ufs-community/ufs-srweather-app/actions/workflows/python_unittests.yaml/badge.svg)](https://github.com/ufs-community/ufs-srweather-app/actions/workflows/python_unittests.yaml)
2121
[![Python functional tests](https://github.com/ufs-community/ufs-srweather-app/actions/workflows/python_func_tests.yaml/badge.svg)](https://github.com/ufs-community/ufs-srweather-app/actions/workflows/python_func_tests.yaml)

devbuild.sh

+13
Original file line numberDiff line numberDiff line change
@@ -417,6 +417,19 @@ else
417417
fi
418418
module list
419419

420+
# Apply patch for sorc/CMakeLists.txt for MacOS arm64/aarch64
421+
if [[ "${PLATFORM}" == "macos" ]]; then
422+
ARCH=$(uname -m)
423+
if [[ "${ARCH}" == arm64 ]] || [[ "${ARCH}" == aarch64 ]]; then
424+
patch1="sorc/patch_macos_arm64_sorc_cmakelists.txt"
425+
if patch -p1 -R --dry-run --silent -d ./sorc -N < ${patch1} 1> /dev/null; then
426+
echo "Patch ${patch1} was already applied";
427+
else
428+
patch -p1 -d ./sorc -N < ${patch1}
429+
fi
430+
fi
431+
fi
432+
420433
mkdir -p ${BUILD_DIR}
421434
cd ${BUILD_DIR}
422435

docs/UsersGuide/source/BackgroundInfo/Introduction.rst

+1-1
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ Since the last release, developers have added a variety of features:
3131

3232
The SRW App v2.2.0 citation is as follows and should be used when presenting results based on research conducted with the App:
3333

34-
UFS Development Team. (2023, Oct. 30). Unified Forecast System (UFS) Short-Range Weather (SRW) Application (Version v2.2.0). Zenodo. https://doi.org/10.5281/zenodo.10015544
34+
UFS Development Team. (2023, Oct. 31). Unified Forecast System (UFS) Short-Range Weather (SRW) Application (Version v2.2.0). Zenodo. https://doi.org/10.5281/zenodo.10015544
3535

3636
User's Guide Organization
3737
============================

docs/UsersGuide/source/BuildingRunningTesting/BuildSRW.rst

+3-4
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ The Unified Forecast System (:term:`UFS`) Short-Range Weather (SRW) Application
1515

1616
To build the SRW App, users will complete the following steps:
1717

18-
#. :ref:`Install prerequisites <HPCstackInfo>`
18+
#. :ref:`Install prerequisites <StackInfo>`
1919
#. :ref:`Clone the SRW App from GitHub <DownloadSRWApp>`
2020
#. :ref:`Check out the external repositories <CheckoutExternals>`
2121
#. :ref:`Set up the build environment and build the executables <BuildExecutables>`
@@ -358,9 +358,8 @@ Additional Details for Building on MacOS or Generic Linux
358358

359359
The SRW App can be built on MacOS and generic Linux machines after the prerequisite software has been installed on these systems (via :term:`HPC-Stack` or :term:`spack-stack`). The installation for MacOS is architecture-independent and has been tested using both x86_64 and M1 chips (running natively). The following configurations for MacOS have been tested:
360360

361-
#. MacBookPro 2019, 2.4 GHz 8-core Intel Core i9 (x86_64), Monterey Sur 12.1, GNU compiler suite v.11.3.0 (gcc, gfortran, g++); mpich 3.3.2 or openmpi/4.1.2
362-
#. MacBookAir 2020, M1 chip (arm64, running natively), 4+4 cores, Big Sur 11.6.4, GNU compiler suite v.11.3.0 (gcc, gfortran, g++); mpich 3.3.2 or openmpi/4.1.2
363-
#. MacBook Pro 2015, 2.8 GHz Quad-Core Intel Core i7 (x86_64), Catalina OS X 10.15.7, GNU compiler suite v.11.2.0_3 (gcc, gfortran, g++); mpich 3.3.2 or openmpi/4.1.2
361+
#. MacBookPro 2019, 2.4 GHz 8-core Intel Core i9 (x86_64), OS Monterey 12.6.1, 32 GB RAM; GNU compiler suite v.12.3.0 (gcc, gfortran, g++); openmpi/4.1.5
362+
#. MacBookAir 2020, M1 chip (arm64, running natively), 4+4 cores, OS Ventura 13.0.1, 16 GB RAM; GNU compiler suite v.12.3.0 (gcc, gfortran, g++); openmpi/4.1.5
364363

365364
Several Linux builds have been tested on systems with x86_64 architectures.
366365

docs/UsersGuide/source/BuildingRunningTesting/ContainerQuickstart.rst

+7-3
Original file line numberDiff line numberDiff line change
@@ -80,19 +80,23 @@ On most Level 1 systems, a container named ``ubuntu20.04-intel-srwapp-release-pu
8080

8181
* - Machine
8282
- File Location
83-
* - Cheyenne/Derecho
83+
* - Cheyenne/Derecho `*`_
8484
- /glade/scratch/epicufsrt/containers
85-
* - Gaea
85+
* - Gaea `*`_
8686
- /lustre/f2/dev/role.epic/containers
8787
* - Hera
8888
- /scratch1/NCEPDEV/nems/role.epic/containers
8989
* - Jet
9090
- /mnt/lfs4/HFIP/hfv3gfs/role.epic/containers
9191
* - NOAA Cloud
9292
- /contrib/EPIC/containers
93-
* - Orion/Hercules
93+
* - Orion/Hercules `*`_
9494
- /work/noaa/epic/role-epic/contrib/containers
9595

96+
.. _`*` :
97+
98+
\* On these systems, container testing shows inconsistent results.
99+
96100
.. note::
97101
* On Gaea, Singularity/Apptainer is only available on the C5 partition, and therefore container use is only supported on Gaea C5.
98102
* The NOAA Cloud containers are accessible only to those with EPIC resources.

docs/UsersGuide/source/BuildingRunningTesting/RunSRW.rst

+89-18
Original file line numberDiff line numberDiff line change
@@ -192,29 +192,100 @@ MacOS requires the installation of a few additional packages and, possibly, an u
192192
Creating the |wflow_env| Environment on Linux and Mac OS
193193
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
194194

195-
On generic Mac and Linux systems, users need to create a conda |wflow_env| environment. The environment can be stored in a local path, which could be a default location or a user-specified location (e.g., ``$HOME/condaenv/venvs/`` directory). (To determine the default location, use the ``conda info`` command, and look for the ``envs directories`` list.) The following is a brief recipe for creating a virtual conda environment on non-Level 1 platforms. It uses the aarch64 (64-bit ARM) Miniforge for Linux and installs into $HOME/conda. Adjust as necessary for your target system.
195+
On generic Mac and Linux systems, users need to create a conda |wflow_env| environment that contains python packages required for running the workflow. Other conda environments may need to be activated for running graphics generation tasks (|graphics_env|) or when testing the AQM/CMAQ (|cmaq_env|). Python packages in these other environments may conflict with those in |wflow_env|. The environments can be stored in a local path, which can be a default location or a user-specified location (e.g., ``$HOME/condaenv/venvs/`` directory). (To determine the default location, use the ``conda info`` command, and look for the ``envs directories`` list.)
196+
These conda environments can be added to the existing python or conda modules.
196197

197-
.. code-block:: console
198+
There are several options available for building virtual conda environments on non-Level 1 platforms. The examples in this section use the aarch64 (64-bit ARM) Miniforge for Linux and install into ``$HOME/conda``. Users should adjust as needed for their target system.
198199

199-
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-aarch64.sh
200-
bash Miniforge3-Linux-aarch64.sh -bfp ~/conda
201-
rm Miniforge3-Linux-aarch64.sh
202-
source ~/conda/etc/profile.d/conda.sh
203-
conda activate
204-
conda install -y conda-build conda-verify
205-
cd path/to/your/workflow-tools/clone
206-
conda build recipe
207-
conda create -y -n workflow_tools -c local workflow_tools
208-
conda activate workflow_tools
200+
**Options:**
209201

210-
In future shells, you can activate and use this environment with:
211202

212-
.. code-block:: console
203+
1) Users can add the following environment .yaml files:
204+
205+
a) workflow_tools.yaml for |wflow_env|
206+
207+
.. code-block:: console
208+
209+
name: workflow_tools
210+
channels:
211+
- conda-forge
212+
- defaults
213+
dependencies:
214+
- python=3.9*
215+
- boto3=1.22*
216+
- black
217+
- f90nml=1.4*
218+
- jinja2=3.0*
219+
- numpy=1.21*
220+
- pylint
221+
- pytest
222+
- pyyaml=6.0*
223+
- tox
224+
225+
b) regional_workflow.yaml for |graphics_env|
226+
227+
.. code-block:: console
228+
229+
name: regional_workflow
230+
channels:
231+
- conda-forge
232+
- defaults
233+
dependencies:
234+
- python=3.9.*
235+
- f90nml
236+
- jinja2
237+
- pyyaml
238+
- scipy
239+
- matplotlib=3.5.2*
240+
- pygrib
241+
- cartopy
242+
243+
c) regional_workflow_cmaq.yaml for |cmaq_env|
244+
245+
.. code-block:: console
246+
247+
name: regional_workflow_cmaq
248+
channels:
249+
- conda-forge
250+
- defaults
251+
dependencies:
252+
- python=3.9.12
253+
- f90nml=1.4*
254+
- jinja2=3.0*
255+
- pyyaml=6.0*
256+
- scipy
257+
- matplotlib
258+
- pygrib
259+
- cartopy
260+
- netcdf4
261+
- xarray
262+
263+
2) Users can instead install Miniforge. This example uses the aarch64 (64-bit ARM) Miniforge for Linux that installs into ``$HOME/conda``. Users should adjust as needed for their target system.
264+
265+
.. code-block:: console
266+
267+
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-aarch64.sh
268+
bash Miniforge3-Linux-aarch64.sh -bfp ~/conda
269+
rm Miniforge3-Linux-aarch64.sh
270+
source ~/conda/etc/profile.d/conda.sh
271+
conda activate
272+
conda install -y conda-build conda-verify
273+
cd path/to/your/workflow-tools/clone
274+
conda build recipe
275+
conda create -y -n workflow_tools -c local workflow_tools
276+
conda activate workflow_tools
277+
278+
In future shells, you can activate and use this environment with:
279+
280+
.. code-block:: console
281+
282+
source ~/conda/etc/profile.d/conda.sh
283+
conda activate workflow_tools
284+
285+
See the `workflow-tools repository <https://github.com/ufs-community/workflow-tools>`__ for additional documentation.
213286

214-
source ~/conda/etc/profile.d/conda.sh
215-
conda activate workflow_tools
287+
3) A third option is to build miniconda3 and create an Lmod modulefile that can be loaded with other modules during the workflow. The module can be added to the user's ``wflow_<platform>.lua`` modulefile, and the environments can be activated or deactivated as needed for a particular workflow task. A repository with full installation instructions, a modulefile template, and environment configuration files can be accessed in `NOAA-EPIC/miniconda3 repository <https://github.com/NOAA-EPIC/miniconda3>`__. Full instructions can be viewed in the `README.md file <https://github.com/NOAA-EPIC/miniconda3/edit/master/README.md>`__.
216288

217-
See the `workflow-tools repository <https://github.com/ufs-community/workflow-tools>`__ for additional documentation.
218289

219290
Modify a ``wflow_<platform>`` File
220291
``````````````````````````````````````
@@ -632,7 +703,7 @@ This can be helpful when conducting multiple experiments with different types of
632703
Plotting Configuration (optional)
633704
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
634705

635-
An optional Python plotting task (PLOT_ALLVARS) can be activated in the workflow to generate plots for the :term:`FV3`-:term:`LAM` post-processed :term:`GRIB2`
706+
An optional Python plotting task (plot_allvars) can be activated in the workflow to generate plots for the :term:`FV3`-:term:`LAM` post-processed :term:`GRIB2`
636707
output over the :term:`CONUS`. It generates graphics plots for a number of variables, including:
637708

638709
* 2-m temperature

docs/UsersGuide/source/CustomizingTheWorkflow/InputOutputFiles.rst

+2-2
Original file line numberDiff line numberDiff line change
@@ -164,8 +164,8 @@ If users wish to modify the fields or levels that are output from the UPP, they
164164
This process requires advanced knowledge of which fields can be output for the UFS Weather Model.
165165

166166
UPP Product Output Tables for the UFS SRW LAM Grid:
167-
* :doc:`3D Native Hybrid Level Fields <tables/SRW_NATLEV_table>`
168-
* :doc:`3D Pressure Level Fields <tables/SRW_PRSLEV_table>`
167+
* :ref:`3D Native Hybrid Level Fields <SRW_NATLEV_table>`
168+
* :ref:`3D Pressure Level Fields <SRW_PRSLEV_table>`
169169

170170
Use the instructions in the `UPP User's Guide <https://upp.readthedocs.io/en/upp-srw-v2.2.0-docs/InputsOutputs.html#control-file>`__ to make modifications to the ``fv3lam.xml`` file and to remake the flat text file, called ``postxconfig-NT-fv3lam.txt`` (default), that the UPP reads.
171171

docs/UsersGuide/source/conf.py

+2
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,8 @@
8989

9090
rst_prolog = """
9191
.. |wflow_env| replace:: ``workflow_tools``
92+
.. |graphics_env| replace:: ``regional_workflow``
93+
.. |cmaq_env| replace:: ``regional_workflow_cmaq``
9294
.. |activate| replace:: ``conda activate workflow_tools``
9395
.. |prompt| replace:: ``(workflow_tools)``
9496
.. |latestr| replace:: v2.2.0

docs/UsersGuide/source/tables/SRW_NATLEV_table.rst

+2
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
:orphan:
22

3+
.. _SRW_NATLEV_table:
4+
35
************************************************************
46
Fields Requested in the UPP Parameter Table for SRW NATLEV
57
************************************************************

docs/UsersGuide/source/tables/SRW_PRSLEV_table.rst

+2
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
:orphan:
22

3+
.. _SRW_PRSLEV_table:
4+
35
**********************************************************
46
Fields Requested in the UPP Parameter Table for SRW PRSLEV
57
**********************************************************

etc/lmod-setup.csh

+9-8
Original file line numberDiff line numberDiff line change
@@ -18,22 +18,23 @@ if ( "$L_MACHINE" != wcoss2 ) then
1818
endif
1919

2020
if ( "$L_MACHINE" == macos ) then
21-
arch=$(uname -m)
22-
[[ "$arch" = arm64 ]] && export ENV="/opt/homebrew/opt/lmod/init/csh"
23-
[[ "$arch" = x86_64 ]] && export ENV="/usr/local/opt/lmod/init/csh"
24-
source $ENV
21+
if ( -d /opt/homebrew/opt/lmod ) then
22+
source /opt/homebrew/opt/lmod/init/csh
23+
else if ( -d /usr/local/opt/lmod ) then
24+
source /usr/local/opt/lmod/init/csh
25+
else
26+
echo "Path for Lmod is unknown, verify whether Lmod is installed"
27+
endif
2528

2629
module purge
2730

2831
else if ( "$L_MACHINE" == linux ) then
29-
setenv ENV "/usr/share/lmod/lmod/init/csh"
30-
source $ENV
32+
source /usr/share/lmod/lmod/init/csh
3133

3234
module purge
3335

3436
else if ( "$L_MACHINE" == singularity ) then
35-
set ENV="/usr/share/lmod/lmod/init/csh"
36-
source $ENV
37+
source /usr/share/lmod/lmod/init/csh
3738

3839
module purge
3940

etc/lmod-setup.sh

+6-4
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/sh
1+
#!/usr/bin/env bash
22

33
if [ $# = 0 ]; then
44
L_MACHINE=${MACHINE}
@@ -26,10 +26,12 @@ fi
2626

2727
if [ "$L_MACHINE" = macos ]; then
2828
arch=$(uname -m)
29-
[[ "$arch" = arm64 ]] && export BASH_ENV="/opt/homebrew/opt/lmod/init/bash"
30-
[[ "$arch" = x86_64 ]] && export BASH_ENV="/usr/local/opt/lmod/init/bash"
29+
if [ "$arch" = arm64 ] || [ "$arch" = aarch64 ]; then
30+
export BASH_ENV="/opt/homebrew/opt/lmod/init/bash"
31+
else
32+
export BASH_ENV="/usr/local/opt/lmod/init/bash"
33+
fi
3134
source $BASH_ENV
32-
3335
module purge
3436

3537
elif [ "$L_MACHINE" = linux ]; then

jobs/JREGIONAL_AQM_ICS

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
#
44
#-----------------------------------------------------------------------

jobs/JREGIONAL_AQM_LBCS

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
#
44
#-----------------------------------------------------------------------

jobs/JREGIONAL_BIAS_CORRECTION_O3

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
#
44
#-----------------------------------------------------------------------

jobs/JREGIONAL_BIAS_CORRECTION_PM25

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
#
44
#-----------------------------------------------------------------------

jobs/JREGIONAL_CHECK_POST_OUTPUT

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
#
44
#-----------------------------------------------------------------------

jobs/JREGIONAL_FIRE_EMISSION

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
#
44
#-----------------------------------------------------------------------

jobs/JREGIONAL_GET_EXTRN_MDL_FILES

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
#
44
#-----------------------------------------------------------------------

jobs/JREGIONAL_MAKE_GRID

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
#
44
#-----------------------------------------------------------------------

jobs/JREGIONAL_MAKE_ICS

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
#
44
#-----------------------------------------------------------------------

jobs/JREGIONAL_MAKE_LBCS

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
#
44
#-----------------------------------------------------------------------

jobs/JREGIONAL_MAKE_OROG

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
#
44
#-----------------------------------------------------------------------

jobs/JREGIONAL_MAKE_SFC_CLIMO

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
#
44
#-----------------------------------------------------------------------

jobs/JREGIONAL_NEXUS_EMISSION

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
#
44
#-----------------------------------------------------------------------

jobs/JREGIONAL_NEXUS_GFS_SFC

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
#
44
#-----------------------------------------------------------------------

jobs/JREGIONAL_NEXUS_POST_SPLIT

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
#
44
#-----------------------------------------------------------------------

jobs/JREGIONAL_PLOT_ALLVARS

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
#
44
#-----------------------------------------------------------------------

0 commit comments

Comments
 (0)