NextGen Forcings Engine Standalone Instructions to Construct a NextGen Forcing File for Calibration Retrospective Formulation Efforts using AORC AWS Data
-
Create an AWS account (AWS Console - Signup (amazon.com)). You will need to put billing information in to grant an account, but It’s free and they won’t charge you unless you physically create a storage bucket.
-
Login to your account, click the top right hand tab and scroll to the “security credentials” tab and access it.
-
Under the AWS IAM credentials tab, scroll down to the “Access keys” portion and click “Create access key”.
-
Once created, you will have an “access key” and a “secret key” credential for your own account. Copy over this information or write it down since we will need to create files and link the keys for the Forcings Engine Python environment.
-
In your Linux environment where the Python environment is stored for the NextGen Forcings Engine implementation, execute the following command to make the proper aws directory for credentials: “mkdir ~/.aws”
-
Go into the newly made aws directory, and create a file called “config”. Insert the following text below and save the file:
-
Now, create a file called “credentials”. Insert the following text logic below with your own access and secret key credentials and save the file:
-
Now you have the aws credentials setup, go ahead and download the NextGen Forcings Engine repo: GitHub - NOAA-OWP/ngen-forcing
-
Go into the ngen-forcing/ ESMF_Mesh_Domain_Configuration_Production directory and setup your Python ESMF Mesh production environment with the following conda commands: “conda env create --name ESMF_Mesh_Production --file=environments.yml”; “conda activate ESMF_Mesh_Production”.
-
Go into the ngen-forcing/ESMF_Mesh_Domain_Configuration_Production directory, where you will execute the following script to translate a NextGen hydrofabric geopackage file into an ESMF mesh netcdf file compliant to implement the NextGen Forcings Engine: python NextGen_hyfab_to_ESMF_Mesh.py /pathway/to/hydrofabric_geopackage.gpkg ./ESMF_Mesh_Hyfab.nc (“conda deactivate” once Python script is finished)
-
Go into ngen-forcing/NextGen_Forcings_Engine_BMI and setup your Python NextGen Forcings Engine environment with the following conda commands: “conda env create --name NextGen_Forcings_Engine --file=environments.yml”; “conda activate NextGen_Forcings_Engine”.
-
Copy over the ngen-forcing/NextGen_Forcings_Engine_BMI/BMI_NextGen_Configs/CONUS/AORC/hydrofabric/config.yml file to the ngen-forcing/NextGen_Forcings_Engine_BMI base directory.
-
Go into the ngen-forcing/NextGen_Forcings_Engine_BMI/config.yml file and edit the following options below:
- InputForcingDirectories: ["”]
- ScratchDir: “/pathway/to/some/scratch/directory”
- Output: 1
- RefcstBDateProc: “Start_Date in YYYYMMDDHHMM”
- GeogridIn: “/pathway/to/ESMF_Mesh_Hyfab.nc”
- Once those options have been modified in the config.yml in step 13, now go into the ngen-forcing/NextGen_Forcings_Engine_BMI/run_bmi_model.py script as the final step before executing the NextGen Forcings Engine Standalone implementation. Edit lines 12-13 with the respective “start_time” and “end_time” you intend on executing the NextGen Forcings Engine to produce a NextGen hydorfabric forcing file input file for calibration/retrospective formulation efforts.
- Execute the standalone NextGen Forcings Engine model with the following python command: mpiexec -n number_of_processors_for_mpi python run_bmi_model.py (Important note for AORC AWS capabilities here! To make this fast/efficient I would suggest supplying at least 10-15 cpus. If it’s a CONUS-scaled hydrofabric simulation, you will want at least 30+ cpus to make this somewhat fast/effective AWS s3 bucket downloading capabilities.
- Once completed, you will find the NextGen Forcings Engine forcing file output configured for the NextGen model engine within the “ScratchDir” pathway you specified in the config.yml file. You’re now good to go moving forward with steps to execute the NextGen Forcings Engine for AORC AWS s3 bucket capabilities for calibration/retrospective formulation efforts under the NextGen framework.