NextGen Forcing Extraction Scripts
Overview of Forcings Extraction Scripts Repo
- This repository contains a series of scripts for each NWM domain subdirectory (CONUS, Alaska, Puerto Rico, Hawaii) that encompasses the required meteorological forcing data products needed for each regional NWMv3.0 operational configuration setup. Each script is a particular meteorological forcing data product that is available to download off the NOMADS server. Availability of each meteorological forcing data product varies, but a user can generally extract at least the last 24 hours of previous data products or forecast cycles available.
Creating Python environment to execute scripts using Anaconda
- Inside the Forcings Extraction Script repo, you should find a environment.yml file that lists out the Python dependencies to execute the Python scripts
- conda env create --name forcing_extraction --file=environment.yml
- conda activate forcing_extraction
Example Executing a Python Forcing Extraction Script
- cd Forcing_Extraction_Scripts/Global
- Make a output directory where the Forcing data from the NOMADS server will be placed (mkdir output).
- Execute the following command below:
Python script arguments overview
- Output directory pathway
- --lookBackHours (This is the amount of hours to look back for extracting forecast cycles or post-processed data for pending on the meteorological data field you’re extracting. Generally this really should go further back than 72 hours, where that is the cap for NOMADs storing previous forecast cycles for meteorological models)
- --cleanBackHours (This just tells the script to clean out forecast model subdirectories within your output directory pathway that may already contain previous forecast cycle data that you want to clean out)
- --lagBackHours (How many hours back do you want to stop the data extraction period for leading up to real time)
Example Output of the Python Forcing Extraction Script
- The script will create a directory pathway equivalent to what is on the NOMADS server and also the structure that is expected within the NextGen Forcings Engine BMI code to search for files within a model application. You will see each file that it pulls of NOMADs server for a given forecast model or dataset as shown below:
- At times, the NOMADS server may be hung up on requests and it takes some time to just download a single file. The Python scripts will continue to wait for a request to download a given meteorological data file for up to ten minutes. If NOMADs does not handle the file request within that ten minute window, then it generally means that the NOMADs server does not have the file available within that meteorological product (which implies an issue on their end or the actual forecast product isn’t available on the server yet). You will see the output statement below if this happens during the extraction process:
