Add QAQC auto reporting tool

[bemcdonnell]

As we work toward making releases, I believe it would be valuable to have a tool that automatically runs the benchmarks and plots/docs the results for the new version. It doesn’t need to be exhaustive for all objects / results - maybe the top differences.

What this tool should use:

PyLaTex

• Python / Matplotlib
• LaTeX

What we should show:

• scatter plots of totalized and average values for all object / results.
• time series plots for object results (top X number of differences)
• numerical fitness between benchmark and new release (some examples: Nash-Sutcliffe, Logaritmic Nash-Sutcliffe, Percent Bias, Kling-Gupta).

[dickinsonre]

@bemcdonnell Good ideas, it might be also good to have the min and max of all elements. It also needs better example files, the bulk of the sample files are from Extran3 and SWMM4 converted to SWMM5. The sample files, IMHO, should have more than a dozen elements. It would be nice to have sample file with everything possible in SWMM5 as a hundred nodes, links and Subcatchments so that it at least looks like a decent sample model.

[bemcdonnell]

@dickinsonre, for previous "official" releases of SWMM, how many benchmark models were used? And what was the process for cherry-picking results to present in the SWMM users manual?

[dickinsonre]

@bemcdonnell We did have about 700 test models for SWMM 5. You can see most of them on my Box account (I just made you and editor). However, most of the files on Box suffer from the same problems as these test files (the user example files do look like real networks as they came from real networks).

Cherry picking, i don't know how Lew picked them but they seem to be most of the Extran3/4 files along with some PCSWMM files. I used the PCSWMM files for SWMM4 testing back in the day.

I just checked and we about 200 test models for SWMM4, 130 hydraulics, 70 Hydrology and Water Quality. SWMM4 also had a nice Graph Block that compared the statistics of the model simulation for selected flows and depths to a user data file. Lew added may more calibration graphs but we lost the nice Statistical comparison possible in SWMM4. It is a good idea to use Python to add that back to the software for testing purposes.

Example 1: pass
Example 2: pass
Example 3: pass
Example 4: pass
Example 5: pass
Example 6: pass
extran 1: pass
extran 10: pass
extran 2: pass
extran 3: pass
extran 4: pass
extran 5: pass
extran 6: pass
extran 7: pass
extran 8a: pass
extran 8b: pass
extran 9: pass
test 1: pass
test 2: pass
test 3: pass
test 4: pass
test 5: pass
user 1: pass
user 2: pass
user 3: pass
user 4: pass
user 5: pass

[dickinsonre]

Hello @bemcdonnell just to follow up on your cherry picking comment. I went to the QA report on the EPA Download site and converted it to a DOC file. The testing Lew Rossman did in the really nice STORM WATER MANAGEMENT MODEL QUALITY ASSURANCE REPORT: Dynamic Wave Flow Routing
involved three sets of data (1) the ten data sets from Extran3 and SWMM4, (2) SWMM5 test data sets (Challenge Test Cases) and (3) User Test Cases (for the most part these come from CDM Cini). I will also mention that all of these cases are included on this GITHUB as part of the NRTEST suite. Does this answer your question about Cherry Picking?

2. Routing Models 6
3. Testing Procedure 13
4. Extran Manual Test Cases 15
5. Challenge Test Cases 53
6. User-Supplied Test Cases 72
7. Summary and Conclusions 110
8. References 111
   Appendix A. SWMM 4 Routing Models 112
   Appendix B. Test Data Sets 115

[bemcdonnell]

@dickinsonre, excellent info! Thanks for diving into that!

[dickinsonre]

@bemcdonnell My pleasure, however, it looks like a lot of work done by Lew to make that document. It will be nice to have a complete automated testing with stats for every node, link, lid, gw, subcatchment etc etc. that can be saved as log files for future backwards and forwards testing.

[michaeltryby]

@dickensonre cool idea!

[michaeltryby]

What if we develop this then morphed it into a feature. Like a calibration report that SWMM users could run as part of their QA. That would save Engineers tons of grunt work. Perhaps help them to be less fixated on changing model results.

[dickinsonre]

Link for Nash-Sutcliffe, Logarithmic Nash-Sutcliffe
https://www.adv-geosci.net/5/89/2005/adgeo-5-89-2005.pdf
Link for Kling-Gupta
http://www.meteo.mcgill.ca/~huardda/articles/gupta09.pdf

[dickinsonre]

I have a conceptual question @bemcdonnell and @michaeltryby say you had a NS value for each node, link and subcatchment. If you have a large model this is still many numbers to view and absorb. How can one come up with one overall index of model comparison? A mean of the NS values even though the node depths and link flows while related are from two different sections of the code? Report the mean node, link and subcatchment NS value?

[bemcdonnell]

@dickinsonre, could always report the top offenders above some threshold in a table as well as MEAN and STD?

[dickinsonre]

A few other references on KGE
https://www.rforge.net/doc/packages/hydroGOF/KGE.html
http://www.smhi.net/hype/wiki/doku.php?id=start:hype_file_reference:info.txt:criteria_equations

[dickinsonre]

Okay, for some reason your @bemcdonnell cherry picking comment got stuck in my head! We have implemented comparison tests for all nodes, links and subcatchments. I will update my dickinsonre GitHub soon after I learn more about GIT. My son was complaining today I am using Git wrong. Thanks for your patience in this matter.