Instructions for Using WRES

The WRES is available as both a webservice, the Central OWP WRES (COWRES) hosted at the National Water Center (NWC) in Alabama, and as a standalone application. When obtained as a standalone, it can either be run as a one-off application, executing an evaluation and then shutting down, or a local-server, a long-running application that can be interacted with through localhost webservice requests. Where appropriate, the different modes of execution will be explained in the instructions below.

The steps to execute an evaluation using the WRES are below.

1. Scope the Evaluation to be Performed

The user must begin by identifying and designing the evaluation to perform. For introductory resources on forecast verification, see Introductory Resources on Forecast Verification.

Planning the Evaluation

(The resources mentioned below are only available to NOAA personnel. We are working to migrate those resources and make them available to the public in the near future.)

For a discussion on how to plan an evaluation, the user is referred to the WRES Training, Part 1: Concepts slide deck and recorded training video (both are available in our Past Training Materials for the Training Conducted on July 15, 2020). In general, the evaluator must answer the questions,

Why do you want to evaluate something?
What do you want to evaluate?
How should the evaluation be conducted?

From there, the evaluator must identify the evaluation data to statistics transformation pipeline:

Statistics <- Pools <- Pairs <- Data

Important points related to this are covered in the slide deck and training video linked above.

Available Evaluation Data

One important aspect of that evaluation plan is identifying the sources of data for both the forecast (or simulation) data to evaluate and the observation (or simulation) data against which to perform the evaluation. In general, evaluation input data can be either

1. requested from a web-service, or
2. available on the local files system with the WRES having read permissions.

The following web services may be accessible options for obtaining input data:

• USGS Observations: The WRES can pull observations directly from the USGS National Water Information System (NWIS) where such observations are available. Instructions for configuration are provided via Example 5 within Complete Examples of Evaluation Declarations. Not only is WRES not responsible for the accuracy of data, but also not responsible for availability of data services nor data availability within those services. One may subscribe to NWIS service announcements at https://listserv.usgs.gov/mailman/listinfo/nwisweb-notification
• Recent NWM data via WRDS Services: If you have access to WRDS services (see note, below), the WRES can read recent (generally, 90 days) NWM data directly from services developed by the OWP WRDS team.
• AHPS Forecasts via WRDS API: If you have access to WRDS services (see note, below), a decades long archive of AHPS forecast data is available to support a WRES evaluation.

NOTE: WRDS services are only accessible from within the NWC network. If you are on that network OR you are using the Central OWP WRES, then you should have access to those services; see the NOAA VLab project WRES User Support wiki for information on how to declare use of WRDS services.

Files provided to the WRES must be in one of the following formats:

• WRES-Compliant CSV Files: The files must follow a specific format, described at Format Requirements for CSV Files.
• CHPS/FEWS PI-timeseries XML files: The files can be gzipped (i.e., *.xml.gz) or tarred and gzipped (see compressed archives, below). However, they may not be gzipped XML files that are then tarred (i.e., a .tar containing .xml.gz is not allowed).
• Fast-Infoset encoded CHPS/FEWS PI-timeseries XML files: The files can be gzipped (i.e., *.fi.gz) or tarred and gzipped (see compressed archives, below). However, they may not be gzipped files that are then tarred (i.e., a .tar containing .fi.gz is not allowed).
• NWS datacard format files: This format is allowed for observed or simulation data only. It is highly recommended that this format be avoided if possible. It insufficiently describes the data contained, therefore requiring specification of declaration that other formats do not.
• WRDS-JSON Format Files: WRDS services will use a JSON format for data interchange. The WRES can also read data formatted following WRDS-JSON from flat files.
• NWM v1.1 - v3.0 compliant netCDF files: To be considered NWM-compliant, the NetCDF must include the expected metadata. For more information, see Format Requirements for NetCDF Files.
• NWM data available on-line: In general, the WRES can read NWM netCDF files from any online location so long as (1) WRES has access and (2) the files are organized identically to what is found in the NOMADS; again, see Configuring the Raw NWM Data Source. For example, it can access data provided through the @para.nomads.ncep.noaa.gov@ website: https://para.nomads.ncep.noaa.gov/pub/data/nccf/com/nwm/para/. NOTE: If you have access to NWC resources, then you can obtain NWM data from the NWC D-Store; see the NOAA VLab WRES User Support project wiki for more information).
• USGS JSON (WaterML): The WRES can read files in USGS-style, JSON, WaterML format. WaterML is described here.

The following compressed archives of files can be read:

• Tarred/Gzipped Archives: The WRES can read archives of tarred/gzipped (e.g., .tgz or .tar.gz) files following any of the formats mentioned above with the exception of raw NWM data.
• Gzipped Data: The WRES can read gzipped (e.g., .gz) files following any of the formats mentioned above with the exception of raw NWM data.

Identify Other Aspects of the Evaluation

Those other aspects include:

• The geographic features to evaluate and whether they should be evaluated separately or pooled into groups. For example, see Pooling geographic features.
• The desired times scale at which the data will be evaluated. The time scale is comprised of both the time period over which a value was derived, or “period”; and how the value was derived over that time period, or “function”. For example, you could evaluate the 24-hour average daily streamflow given data that is 6-hour instantaneous streamflow. You could also evaluate an accumulated precipitation amount over a 24-hour period calculated from 6-hour accumulated precipitation amounts. Optionally, rescaling may be performed using a fixed interval that is bounded by one or two month-days, such as 1 April through 31 July or 90 days beginning on 1 April. For further guidance, see Time Scale and Rescaling Time Series.
• Whether a baseline forecast system, against which the target forecast system is to be compared, will be included in an evaluation. Such a baseline forecast can have implications for which (observation, forecast) pairs are included in an evaluation, since ensuring an apples-to-apples comparison is important. If pairs for different forecast systems span different time periods or include different gaps within the same time period, then that could lead to an invalid comparison that is not apples-to-apples.
• How the verification (observation, forecast) pairs will be pooled for calculating metrics. By default, metrics are calculated for all provided pairs, regardless of lead time. However, most often statistics will need to be computed for each lead time, independently, since there is typically a strong dependence between forecast quality and lead time. Pooling windows can also be defined relative to issued time (i.e., forecast basis time or T0), allowing for a rolling windows analysis or a seasonal analysis; as well as relative to thresholds, allowing for evaluation of high flow forecasts.
• The metrics desired. For a list of available metrics, see the List of Metrics Available.
• Whether to calculate the sampling uncertainties. For further details see Sampling uncertainty assessment.
• The outputs desired. Outputs can include ASCII Comma Separated Value (CSV) files; 2-D chart PNG files; pairs CSV files; and NetCDF files. See 5. Examine the WRES Output Verification Statistics

2. Prepare the Data (Optional)

The user must identify the data necessary for their evaluation. There are four types of data that can be supplied to WRES for an evaluation:

• The observed or “left” data sources: Typically, observations or simulations against which simulations or forecasts are evaluated.
• The predicted or “right” data sources: Typically, simulations or forecasts that are to be evaluated.
• The baseline data sources: Typically, simulations or forecasts against which the predicted data will be compared to, for example, calculate skill scores.
• other: These are files or web resources that do not contain evaluation data, but rather other data or information required for an evaluation. For example, thresholds can be provided in CSV file that the evaluation declaration points to.

The data for all sources can either be,

(1) a web service to which the WRES has access, such as USGS NWIS or WRDS (if running in the NWC network); (2) a file local to the WRES instance and to which it has read permissions; (3) a file placed online in a location the WRES can access; (4) a file posted directly to the COWRES instance (if a COWRES instance is being used; see below).

Posting data directly to the COWRES

If the COWRES is to be used to perform the evaluation, then data can be posted directly to the COWRES to support that evaluation. For more information, see the NOAA VLab WRES User Support project wiki (to which you will have access if you can access the COWRES) for more information.

3. Prepare the WRES Project Declaration

Declaration of an evaluation project is described in the wiki, Declaration language. For users who intend to make use of the COWRES hosted at the NWC, the WRES Graphical User Interface (WRES GUI) is highly recommended to help with declaring the evaluation. For more information, see the WRES Cookbook available in the NOAA VLab WRES User Support project wiki.

Checking a declaration before use

UNDER CONSTRUCTION!

4. Execute the WRES Project Through the Central OWP WRES

UNDER CONSTRUCTION! COWRES needs to point the reader to the WRES User Support wiki. Otherwise, provide instructions here for one-time run, or point to local server for that style of run. Something like that.

5. Examine the WRES Output Verification Statistics

With the output gathered, it must then be interpreted. The user should also identify if different or additional outputs are required; in other words, if another execution of WRES is needed.

To support this examination, the WRES includes a variety of graphical and numerical output formats for statistics, including:

• Comma Separated Values (CSV; declaration: csv2; file extensions: .csv.gz and .csvt; see Output Format Description for CSV2)
• Network Common Data Form (NetCDF; declaration: netcdf2; file extension: .nc)
• Portable Network Graphics (PNG; declaration png or graphic; file extension: .png)
• Scalable Vector Graphics (SVG; declaration svg; file extension: .svg)
• Protocol Buffers (Protobuf; declaration protobuf; file extension: .pb3)

6. Repeat Previous Steps as Needed

If another execution of WRES is required, then return to Step 3, as appropriate. If additional data is required, return to Step 2.

7. Final Clean Up

Clean up generally consists of removing any intermediary files no longer needed once an evaluation is complete. This is particularly important if a user is not running an evaluation on their local system, but, instead, employing a web service such as the COWRES. In that case, they should employ whatever mechanism is provided to tell the service that they are done with their evaluation, and inputs (where appropriate) and outputs can be removed. For the COWRES in particular, see the NOAA VLab WRES User Support project wiki for more information.

Requesting user Support

To request a new feature related to an evaluation to perform or report a bug, post a ticket through the GitHub project, https://github.com/NOAA-OWP/wres/issues/new.

NWS users, only, should make any requests for assistance, report bugs, or request new features by a ticket in the NOAA VLab WRES User Support project. Post a ticket and the WRES team will respond as quickly as possible.