Preview of development efforts

CHANGELOG.md

@@ -2,12 +2,11 @@ All notable changes to this project will be documented in this file.
 We follow the [Semantic Versioning 2.0.0](http://semver.org/) format.
 
 
-## Pre-Release 0.0.0 2020-09-14
+## x.y.z - YYYY-MM-DD
 
 ### Added
 
-- Initial configuration management
-- Initial calibration workflow (with many TODO's)
+- Lorem ipsum dolor sit amet
 
 ### Deprecated
 

README.md

@@ -1,55 +1,52 @@
-# NGen Calibration
-
-**Description**:  Supporting code/workflows for automated calibration of [NGen](https://github.com/noaa-owp/ngen) Formulations using Dynamic Dimensioned Search (DDS)
-
-  - **Technology stack**: Python based workflow for generating [NGen](https://github.com/noaa-owp/ngen) parameter formulation permutations and running these through the [NGen](https://github.com/noaa-owp/ngen) framework driver.
-  - **Status**:  This work is in pre-release development, for more details see the [CHANGELOG](CHANGELOG.md).
-
-## Dependencies
-
-Unit testing done with [pytest](https://github.com/pytest-dev/pytest).  See [requirements.txt](requirements.txt) for other specific python dependencies.
-
-## Installation
-
-`TODO` [INSTALL](INSTALL.md) document.
-
-## Configuration
-
-`TODO`
-
-## Usage
-
-`TODO`
-
-## How to test the software
-
-
-Install `pytest` and other python dependencies (`pip install -r python/requirements.txt`)
-Or create a virtual environment:
+## NextGen_Model_Calibration
+>>>>>>> d0a6192 (Add readme and changlog info from development work)
+
+## __Context__
+<font size="4"> This branch currently contains the scripts and files from NextGen_Model_Calibration package and [ngen-cal](https://github.com/NOAA-OWP/ngen-cal/) repo. Detailed information on [ngen-cal](https://github.com/NOAA-OWP/ngen-cal/) can be found on the master branch. This page focuses on the introduction for NextGen_Model_Calibration. </font> 
+
+## __NextGen_Model_Calibration__
+<font size="4"> NextGen_Model_Calibration is a model-agnostic Python software supporting automatic calibration of a variety of Next Generation Water Modeling Framework (NextGen) formulations. </font> 
+
+### Main Features
+- <font size="4"> Creates a plethora of input data and configuration files. </font>
+- <font size="4"> Calibrates parameters for different NextGen model and model components such as Conceptual Functional Equivalent (CFE), TOPMODEL, Noah-OWP-Modular, Lumped Arid/Semi-arid Model (LASAM), soil freeze-thaw model (SFT) and soil moisture profile (SMP). </font>
+- <font size="4"> Implements three parameter optimization algorithms, including dynamically dimensioned search (DDS), particle swarm optimization (PSO) and grey wolf optimizer (GWO). </font>
+- <font size="4"> Supports three calibration strategies, including uniform, independent and explicit. </font>
+- <font size="4"> Provides statistical evaluation tool to calculate a number of metrics to measure model performance.
+- <font size="4"> Installs interface to choose objective function from the statistical tool, such as KGE, NSE, MAE, RMSE, RSR, correlation coefficient, etc. </font>
+- <font size="4"> Provides visualization tool to plot different output files from calibration and validation runs.</font>
+
+### Subpackages
+- <font size="4">__createInput__ (Source code in _python/createInput_): This subpackage generates the required input data and configuration files for the specified formulation and basin. </font>
+- <font size="4">__ngen.cal__ (Source code in _python/runCalibValid/ngen_cal_): This subpackge executes simulation through model engine, evaluates streamflow performance and optimizes model parameters. It also performs validation runs using the default and the calibrated best parameter set, respectively. </font>
+- <font size="4">__ngen.config__ (Source code in _python/runCalibValid/ngen_conf_): This subpackage parses and validates realization configurations of a wide range of NextGen model and model components. </font>
+
+### Installation
+<font size="4"> Create and activate the Python virtual environment </font>
+
+``` bash
+python3 -m venv venv-cal 
+source venv-cal/bin/activate
 ```
-mkdir venv
-virtualenv ./venv
-source ./venv/bin/activate
-pip install -r python/requirements.txt
+<font size="4"> Grab source code from the [PR branch](https://github.com/NOAA-OWP/ngen-cal/tree/4cb1c44280e56fd0ac640f0d4e6b5cc57fb3dbd8)</font>
+``` bash
+git clone https://github.com/NOAA-OWP/ngen-cal.git
+cd ngen-cal
+git fetch origin pull/123/head:123
+git checkout 123
 ```
-Then use `pytest` to run the tests:
-`pytest --log-cli-level 0 python/ngen-cal/test/`
-## Known issues
-## Known issues
-
-
-## Getting help
-
-If you have questions, concerns, bug reports, etc, please file an issue in this repository's Issue Tracker.
-
-## Getting involved
-
-See [CONTRIBUTING](CONTRIBUTING.md), or open an issue to start a conversation!
-----
-
-## Open source licensing info
-1. [TERMS](TERMS.md)
-2. [LICENSE](LICENSE)
-
-
-----
+<font size="4"> Install each subpackage into the virtual environment </font>
+```bash
+cd python/createInput
+pip install .
+cd python/runCalibValid/ngen_cal
+pip install .
+cd python/runCalibValid/ngen_conf
+pip install .
+```
+### Documentation
+<font size="4"> This calibration tool provides an easily operated interface to perform model parameter optimization. The detailed instructions on installation, configuration and execution of calibration workflow are described in the [User's Guide](https://github.com/NOAA-OWP/ngen-cal/blob/b5b056e4af0e7d5952b19d574d03f8638e31a0c7/doc/NextGen_Model_Calibration_User_Guide.pdf). This document also describes preparation of data and files, as well as explains the scripts used in each subpackage. </font>
+
+### Examples
+<font size="4"> A [shell script](https://github.com/NOAA-OWP/ngen-cal/blob/8541009f70a23e7f50ac6e0ea712c00704a410dc/examples/run_calibration.sh) in examples directory demonstrates the process to generate input file for the specified basin, and execute calibration and validation job runs using the functionalities in this tool. </font>
+

examples/run_calibration.sh

@@ -0,0 +1,49 @@
+#!/bin/bash
+# ------------------------------------------------
+# This script automates the entire calibration process
+# from generating input to executing calibration and validation runs.
+# User needs to specify basin, workdir and scriptdir.
+
+# workdir is the concatenation of entries for options main_dir, objective_function,  
+# optimization_algorithm, and model specified in input.config:
+# workdir=main_dir/objective_function_optimization_algorithm/model  
+
+# Author: Xia Feng
+# ------------------------------------------------
+
+# Activate virtual environment
+source venv-cal/bin/activate 
+python=`which python3`
+
+# Working Directory 
+basin='01010101'
+workdir='working_dir' 
+
+# Create input data and files 
+pyscript1='scriptdir/createInput/src/create_input.py'
+config_file_input='scriptdir/python/createInput/configs/input.config'
+echo 'Creating Input Data and Files'
+$python $pyscript1 $config_file_input 
+echo 'Input Data and Files are Generated'
+
+# Execute calibration run
+pyscript2='scriptdir/runCalibValid/calibration.py'
+config_file_calib=$workdir$basin'/Input/'$basin'_config_calib.yaml'
+echo 'Running Calibration Simulation'
+$python $pyscript2 $config_file_calib >>'calib_run_out'
+echo 'Calibration Run Is Completed'
+
+# Execute validation run
+pyscript3='scriptdir/runCalibValid/validation.py'
+config_file_control=$workdir$basin'/Output/Validation_Run/'$basin'_config_valid_control.yaml'
+config_file_best=$workdir$basin'/Output/Validation_Run/'$basin'_config_valid_best.yaml'
+config_file_valid_run=( $config_file_control $config_file_best )
+run_type=( 'control' 'best' )
+i=0
+while [ $i -le 1 ]
+do
+    echo 'Running Validation' ${run_type[$i]^} 'Simulation'
+    $python $pyscript3 ${config_file_valid_run[$i]} >>'valid_run_out'
+    i=$(($i+1))
+done
+echo 'Validation Run Is Completed'

python/createInput/CHANGELOG.md

@@ -0,0 +1,21 @@
+All notable changes to this project will be documented in this file.
+We follow the [Semantic Versioning 2.0.0](http://semver.org/) format.
+
+
+## x.y.z - YYYY-MM-DD
+
+### Added
+
+- Lorem ipsum dolor sit amet
+
+### Deprecated
+
+- Nothing.
+
+### Removed
+
+- Nothing.
+
+### Fixed
+
+- Nothing.

python/createInput/CONTRIBUTING.md

@@ -0,0 +1,32 @@
+# Guidance on how to contribute
+
+> All contributions to this project will be released to the public domain.
+> By submitting a pull request or filing a bug, issue, or
+> feature request, you are agreeing to comply with this waiver of copyright interest.
+> Details can be found in our [TERMS](TERMS.md) and [LICENSE](LICENSE).
+
+
+There are two primary ways to help:
+ - Using the issue tracker, and
+ - Changing the code-base.
+
+
+## Using the issue tracker
+
+Use the issue tracker to suggest feature requests, report bugs, and ask questions.
+This is also a great way to connect with the developers of the project as well
+as others who are interested in this solution.
+
+Use the issue tracker to find ways to contribute. Find a bug or a feature, mention in
+the issue that you will take on that effort, then follow the _Changing the code-base_
+guidance below.
+
+
+## Changing the code-base
+
+Generally speaking, you should fork this repository, make changes in your
+own fork, and then submit a pull request. All new code should have associated
+unit tests that validate implemented features and the presence or lack of defects.
+Additionally, the code should follow any stylistic and architectural guidelines
+prescribed by the project. In the absence of such guidelines, mimic the styles
+and patterns in the existing code-base.

python/createInput/INSTALL.md

@@ -0,0 +1,3 @@
+# Installation instructions
+
+Detailed instructions on how to install, configure, and get the project running.

python/createInput/LICENSE

@@ -0,0 +1,7 @@
+“Software code created by U.S. Government employees is not subject to copyright
+in the United States (17 U.S.C. §105). The United States/Department of Commerce
+reserve all rights to seek and obtain copyright protection in countries other
+than the United States for Software authored in its entirety by the Department
+of Commerce. To this end, the Department of Commerce hereby grants to Recipient
+a royalty-free, nonexclusive license to use, copy, and create derivative works
+of the Software outside of the United States.”

python/createInput/README.md

@@ -0,0 +1,12 @@
+## CreateInput
+
+<font size="4"> CreateInput is a Python package to generate the data and files needed for performing the model parameter calibration and validation in the NextGen modeling framework. </font>
+
+## Files in this Subpackage 
+- <font size="4">___configs_ Directory__ 
+    - <font size="4">_input.config_: This input configuration file contains all kinds of options related to the selected basin, working directory, formulation, optimization algorithm, objective function, iteration number, simulation time period, hydrofabric and forcing data, executable and libraries, etc. </font>
+- <font size="4">___src_ Directory__
+    - <font size="4">_ginputfunc.py_: Contains various functions to create crosswalk file, generates or process BMI initial configuration files, as well as create t-route configuration file, model realization configuration file, and calibration configuration file. </font>
+  - <font size="4">_create_input.py_: Main script to read configuration file _input.config_ and call functions in _ginputfunc.py_ to generate the required input data and files. </font>
+
+

python/createInput/SECURITY.md

@@ -0,0 +1,27 @@
+# Security Policy
+
+There MUST be no unpatched vulnerabilities of medium or higher severity that have been publicly known for more than 60 days. 
+
+The vulnerability must be patched and released by the project itself (patches may be developed elsewhere). A vulnerability becomes publicly known (for this purpose) once it has a CVE with publicly released non-paywalled information (reported, for example, in the <a href="https://nvd.nist.gov/">National Vulnerability Database</a>) or when the project has been informed and the information has been released to the public (possibly by the project). A vulnerability is considered medium or higher severity if its <a href="https://www.first.org/cvss/" >Common Vulnerability Scoring System (CVSS)</a> base qualitative score is medium or higher. In CVSS versions 2.0 through 3.1, this is equivalent to a CVSS score of 4.0 or higher. Projects may use the CVSS score as published in a widely-used vulnerability database (such as the <a href="https://nvd.nist.gov">National Vulnerability Database</a>) using the most-recent version of CVSS reported in that database. Projects may instead calculate the severity themselves using the latest version of <a href="https://www.first.org/cvss/">CVSS</a> at the time of the vulnerability disclosure, if the calculation inputs are publicly revealed once the vulnerability is publicly known. <strong>Note</strong>: this means that users might be left vulnerable to all attackers worldwide for up to 60 days. This criterion is often much easier to meet than what Google recommends in <a href="https://security.googleblog.com/2010/07/rebooting-responsible-disclosure-focus.html">Rebooting responsible disclosure</a>, because Google recommends that the 60-day period start when the project is notified _even_ if the report is not public. Also note that this badge criterion, like other criteria, applies to the individual project. Some projects are part of larger umbrella organizations or larger projects, possibly in multiple layers, and many projects feed their results to other organizations and projects as part of a potentially-complex supply chain. An individual project often cannot control the rest, but an individual project can work to release a vulnerability patch in a timely way. Therefore, we focus solely on the individual project's response time.  Once a patch is available from the individual project, others can determine how to deal with the patch (e.g., they can update to the newer version or they can apply just the patch as a cherry-picked solution).
+
+The public repositories MUST NOT leak any valid private credential (e.g., a working password or private key) that is intended to limit public access.
+
+## Supported Versions
+
+Use this section to tell people about which versions of your project are
+currently being supported with security updates.
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 5.1.x   | :white_check_mark: |
+| 5.0.x   | :x:                |
+| 4.0.x   | :white_check_mark: |
+| < 4.0   | :x:                |
+
+## Reporting a Vulnerability
+
+Use this section to tell people how to report a vulnerability.
+
+Tell them where to go, how often they can expect to get an update on a
+reported vulnerability, what to expect if the vulnerability is accepted or
+declined, etc.

python/createInput/TERMS.md

@@ -0,0 +1,19 @@
+## Disclaimer
+
+This repository is a scientific product and is not official communication of the National Oceanic and Atmospheric Administration, or the United States Department of Commerce. All NOAA GitHub project code is provided on an 'as is' basis and the user assumes responsibility for its use. Any claims against the Department of Commerce or Department of Commerce bureaus stemming from the use of this GitHub project will be governed by all applicable Federal law. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation or favoring by the Department of Commerce. The Department of Commerce seal and logo, or the seal and logo of a DOC bureau, shall not be used in any manner to imply endorsement of any commercial product or activity by DOC or the United States Government.
+
+[NOAA GitHub Policy](https://github.com/NOAAGov/Information)
+
+### Other Information
+
+Unless expressly stated otherwise, the person who associated a work with
+this deed makes no warranties about the work, and disclaims liability for
+all uses of the work, to the fullest extent permitted by applicable law.
+When using or citing the work, you should not imply endorsement by the
+author or the affirmer.
+
+## Exceptions
+
+_Source code or other assets that are excluded from the TERMS should be listed
+here. These may include dependencies that may be licensed differently or are
+not in the public domain._

python/createInput/configs/calib_params_CFE_NOM.txt

@@ -0,0 +1,19 @@
+param            min       max        init       model
+b                2         15         4.05       CFE  
+satpsi           0.03      0.955      0.355      CFE  
+satdk            1e-7      0.000726   3.38e-06   CFE  
+maxsmc           0.16      0.59       0.439      CFE  
+refkdt           0.1       4.0        1.0        CFE  
+slope            0.0       1.0        0.1        CFE  
+max_gw_storage   0.01      0.25       0.05       CFE  
+expon            1.0       8.0        3.0        CFE  
+Cgw              1.8e-06   0.0018     1.8e-05    CFE  
+Klf              0.0       1.0        0.01       CFE  
+Kn               0.0       1.0        0.03       CFE  
+RSURF_EXP        1.0       6.0        5.0        NoahOWP 
+CWP              0.09      0.36       0.18       NoahOWP 
+MP               3.6       12.6       9.0        NoahOWP 
+VCMX25           24        112        52.2       NoahOWP 
+MFSNO            0.5       4          2          NoahOWP 
+RSURF_SNOW       0.136     100        50         NoahOWP 
+SCAMAX           0.7       1          0.9        NoahOWP