Skip to content

Commit

Permalink
v2.0.7.9031
Browse files Browse the repository at this point in the history
* refactor: Update Shiny app
    + typos
    + indents
    + replace some tables with figures
  • Loading branch information
leppott committed Mar 1, 2024
1 parent 1df0004 commit 96a36ae
Show file tree
Hide file tree
Showing 29 changed files with 145 additions and 997 deletions.
2 changes: 1 addition & 1 deletion DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Package: ContDataQC
Title: Quality Control (QC) of Continous Monitoring Data
Version: 2.0.7.9031
Version: 2.0.7.9032
Authors@R: c(
person("Erik W", "Leppo", email="[email protected]",role=c("aut","cre")),
person("Ann","Roseberry Lincoln", role="ctb"),
Expand Down
11 changes: 10 additions & 1 deletion NEWS
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,21 @@ NEWS-ContDataQC

<!-- NEWS.md is generated from NEWS.Rmd. Please edit that file -->

#> Last Update: 2024-02-27 14:17:40.109261
#> Last Update: 2024-03-01 13:34:32.498829

# Version History

## v2.0.7.9031

2024-03-01

- refactor: Update Shiny app
- typos
- indents
- replace some tables with figures

## v2.0.7.9031

2024-02-27

- refactor: Update Shiny app text and tables, Issue \#156
Expand Down
11 changes: 10 additions & 1 deletion NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,21 @@ NEWS-ContDataQC

<!-- NEWS.md is generated from NEWS.Rmd. Please edit that file -->

#> Last Update: 2024-02-27 14:17:40.109261
#> Last Update: 2024-03-01 13:34:32.498829

# Version History

## v2.0.7.9031

2024-03-01

- refactor: Update Shiny app
- typos
- indents
- replace some tables with figures

## v2.0.7.9031

2024-02-27

- refactor: Update Shiny app text and tables, Issue \#156
Expand Down
8 changes: 8 additions & 0 deletions NEWS.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,14 @@ cat(paste0("Last Update: ",Sys.time()))

# Version History

## v2.0.7.9031
2024-03-01

* refactor: Update Shiny app
+ typos
+ indents
+ replace some tables with figures

## v2.0.7.9031
2024-02-27

Expand Down
2 changes: 1 addition & 1 deletion inst/shiny-examples/ContDataQC/global.R
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ library(shinyjs)
# Sys.setenv(PATH = paste(Sys.getenv("PATH"), "C:\\Rtools\\bin", sep = ";"))

# Version Number
version <- "2.0.7.9031"
version <- "2.0.7.9032"

#Maximum individual file size that can be uploaded is 70 MB
options(shiny.maxRequestSize = 70 * 1024^2)
Expand Down
2 changes: 1 addition & 1 deletion inst/shiny-examples/ContDataQC/rmd/App_1b_TestData.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,4 @@ You’ll see two sets of folders. Within each folder are example input and outpu

* **miniDOT_concatenate** - When data are initially downloaded from miniDOT sensors, there are separate .txt files for each day (in this example, 325 individual files). Go to the **Automated Reformat – miniDOT** tab and run the **Concatenate** function to combine them all into one file.

* **miniDOT_reformat** – run the combined file through the Reformat function. The function reformats the file so that it is ready to run through the QC report function. Next, go to the Main **Functions-Import Files**s tab, import the reformatted file, run it through the ‘QC raw data’ function, and check the flagged data (edit if needed).
* **miniDOT_reformat** – run the combined file through the Reformat function. The function reformats the file so that it is ready to run through the QC report function. Next, go to the Main **Functions-Import Files** tab, import the reformatted file, run it through the ‘QC raw data’ function, and check the flagged data (edit if needed).
4 changes: 2 additions & 2 deletions inst/shiny-examples/ContDataQC/rmd/App_1c_FAQ.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ if(boo_DEBUG==TRUE){

**Failure to run:** When the Shiny app fails to run, the screen will gray out and the message 'disconnected from the server' will appear. The problem typically stems from a formatting issue with the input file, in particular the Date/Time field. Check your input file, refresh the app and try again. If you still have problems, click the 'contact us' link at the bottom of this page and request assistance. Be prepared to share a copy of your input file(s) and the error message.

**Speed:** If you have a slow internet connection, you can run ContDataQC as a Shiny app on your local computer and it will likely be faster. Running it locally requires that you have R software and the ContDataQC R package installed on your computer, which can be downloaded from GitHub <a class="menu__link" href="https://github.com/leppott/ContDataQC" target="_blank">GitHub<span class="usa-tag external-link__tag" title="Exit EPA Website">
**Speed:** If you have a slow internet connection, you can run ContDataQC as a Shiny app on your local computer and it will likely be faster. Running it locally requires that you have R software and the ContDataQC R package installed on your computer, which can be downloaded from <a class="menu__link" href="https://github.com/leppott/ContDataQC" target="_blank">GitHub<span class="usa-tag external-link__tag" title="Exit EPA Website">
<span aria-hidden="true">Exit</span>
<span class="u-visually-hidden"> Exit EPA Website</span>
</span>
Expand All @@ -42,7 +42,7 @@ if(boo_DEBUG==TRUE){
**Usability on phones:** Mobile use of this app is possible although the screen size of a phone may make it impractical.

**Data summary and visualization:** The ContDataSumViz app has more summary and visualization options.
Shiny app (beta version): <a class="menu__link" href="https://contdataqcsumviz_containerized_stg.app.cloud.gov/" target="_blank">https://contdataqcsumviz_containerized_stg.app.cloud.gov/<span class="usa-tag external-link__tag" title="Exit EPA Website">
Shiny app (beta version): <a class="menu__link" href="https://dmap-contdataqcsumviz.app.cloud.gov/" target="_blank">https://dmap-contdataqcsumviz.app.cloud.gov/<span class="usa-tag external-link__tag" title="Exit EPA Website">
<span aria-hidden="true">Exit</span>
<span class="u-visually-hidden"> Exit EPA Website</span>
</span>
Expand Down
41 changes: 33 additions & 8 deletions inst/shiny-examples/ContDataQC/rmd/App_1d_Tips.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -24,52 +24,77 @@ Below are tips and links to resources from the Regional Monitoring Network (RMN)
## Site visits

It helps speed up the Quality Control (QC) process if you follow a checklist during site visits and document anything that might affect the quality of the data (e.g., sensor out of water or buried in sediment, beaver activity, low battery).

* **Resource:** [Site visit checklist](SiteVisitChecklist.zip) [ZIP]

## Sensor configuration

Some people have had problems with air and water sensors being out of sync (
e.g., one records at 11:00 and the other records at 11:07). If you are deploying air and water sensors at a site, make sure you configure them so that they are recording at the same time. This will make data processing faster and easier.
e.g., one records at 11:00 and the other records at 11:07). If you are deploying
air and water sensors at a site, make sure you configure them so that they are
recording at the same time. This will make data processing faster and easier.

* **Resource:** [Onset HOBO configuration tips](HOBO_ConfigLaunch_20170803.pdf) [PDF]

Another issue that sometimes occurs is data overlap (where more than one file has measurements covering part of the same time periods). Make sure you clear the sensor's memory when you download data and relaunch sensors to avoid overlapping data.

* **Resource:** [Onset HOBO download and relaunch tips](HOBO_DataDownload_20170823.pdf) [PDF]

## QC workflow

There are generally two scenarios: 1) users QC their data after each download and work with one file per site at a time; or 2) users have a backlog of data that cover multiple deployment periods that they aggregate into one file before performing QC.
There are generally two scenarios: 1) users QC their data after each download
and work with one file per site at a time; or 2) users have a backlog of data
that cover multiple deployment periods that they aggregate into one file before
performing QC.

* **Resources:** Suggested workflows for each scenario:
+ [Single file](Workflow_QC_report_20220824.pdf) [PDF]
+ [Multiple files](Workflow_DataPileup_20220824.pdf) [PDF]

## Data edits

When reviewing the QC reports, it is important to be consistent in how you handle flagged data. Here is an example of an approach -
When reviewing the QC reports, it is important to be consistent in how you
handle flagged data. Here is an example of an approach -

* If you are certain a data point is erroneous, delete the measurement and flag as ‘F’. If you are not sure, flag the data as ‘S’ and do not delete. Let the people using the data decide whether to remove questionable measurements from their analysis.
* If you are certain a data point is erroneous, delete the measurement and flag
as ‘F’. If you are not sure, flag the data as ‘S’ and do not delete. Let the
people using the data decide whether to remove questionable measurements from
their analysis.

* Document that you checked each point flagged as ‘F’ and ‘S’ by adding a note
or data qualifier to the Comment column.

* Document that you checked each point flagged as ‘F’ and ‘S’ by adding a note or data qualifier to the Comment column.
* **Resource:** [List of example data qualifiers](DataQualifiers_20220210.xlsx) [XLSX]

* Leave missing data cells as is (vs. deleting)


## Accuracy checks

Accuracy checks are comparisons of discrete or in situ measurements taken in the lab and/or in the field with sensor measurements from the closest date/time. The difference between the sensor and discrete measurements should be within the accuracy quoted by the manufacturer (e.g., ±0.2°C if you are using the Onset HOBO proV2 sensor).
Accuracy checks are comparisons of discrete or in situ measurements taken in the
lab and/or in the field with sensor measurements from the closest date/time. The
difference between the sensor and discrete measurements should be within the
accuracy quoted by the manufacturer (e.g., ±0.2°C if you are using the Onset
HOBO proV2 sensor).

* **Resource:** [Example accuracy check worksheet](EXAMPLE_AccuracyCheckWkst.xlsx) [XLSX]

## Visual checks of time series plots

Visual checks of time series plots are an important part of the QC process. Some issues, such as dewatering, sediment burial, ice cover and beaver activity, show fairly common patterns, and we've been compiling examples of these patterns to help people recognize potential QC issues with their data.
Visual checks of time series plots are an important part of the QC process. Some issues, such as dewatering, sediment burial, ice cover and beaver activity, show fairly common patterns, and we've been compiling examples of these patterns to
help people recognize potential QC issues with their data.

* **Resource:** [Visual checks](PlotQC_WatchList_20220824.pdf) [PDF]

## Checking sensor data against other data sources

Some partners have been downloading data from nearby weather stations and USGS gages, as well as modeled air temperature and precipitation data from sources such as Daymet (<a class="menu__link" href="https://daymet.ornl.gov/getdata" target="_blank">https://daymet.ornl.gov/getdata<span class="usa-tag external-link__tag" title="Exit EPA Website">
Some partners have been downloading data from nearby weather stations and USGS
gages, as well as modeled air temperature and precipitation data from sources
such as Daymet (<a class="menu__link" href="https://daymet.ornl.gov/getdata" target="_blank">https://daymet.ornl.gov/getdata<span class="usa-tag external-link__tag" title="Exit EPA Website">
<span aria-hidden="true">Exit</span>
<span class="u-visually-hidden"> Exit EPA Website</span>
</span>
</a>), and comparing those data to sensor measurements as part of their QC process.

* **Resource:** [Daymet or USGS gage check](Daymet_Wx_Gage.zip) [ZIP]

4 changes: 2 additions & 2 deletions inst/shiny-examples/ContDataQC/rmd/App_1f_RelatedApps.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Below are links to some additional Shiny apps and/or R code that may also be use

### ContDataSumViz, for summarizing and visualizing QC’d continuous sensor data

Shiny app (beta version): <a class="menu__link" href="https://contdataqcsumviz_containerized_stg.app.cloud.gov/" target="_blank">https://contdataqcsumviz_containerized_stg.app.cloud.gov/<span class="usa-tag external-link__tag" title="Exit EPA Website">
Shiny app (beta version): <a class="menu__link" href="https://dmap-contdataqcsumviz.app.cloud.gov/" target="_blank">https://dmap-contdataqcsumviz.app.cloud.gov/<span class="usa-tag external-link__tag" title="Exit EPA Website">
<span aria-hidden="true">Exit</span>
<span class="u-visually-hidden"> Exit EPA Website</span>
</span>
Expand Down Expand Up @@ -65,7 +65,7 @@ Shiny app: <a class="menu__link" href="https://nalms.shinyapps.io/LakeMonitoR/"
</span>
</a>

GitHub: <a class="menu__link" https://github.com/leppott/LakeMonitoR/" target="_blank">https://github.com/leppott/LakeMonitoR/<span class="usa-tag external-link__tag" title="Exit EPA Website">
GitHub: <a class="menu__link" href="https://github.com/leppott/LakeMonitoR/" target="_blank">https://github.com/leppott/LakeMonitoR/<span class="usa-tag external-link__tag" title="Exit EPA Website">
<span aria-hidden="true">Exit</span>
<span class="u-visually-hidden"> Exit EPA Website</span>
</span>
Expand Down
45 changes: 4 additions & 41 deletions inst/shiny-examples/ContDataQC/rmd/App_2a1_HOBO.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -85,49 +85,12 @@ Running the HOBOware reformat function on a HOBO U20 water level logger file.

**Before**

```{r}
library(readxl)
library(knitr)
library(kableExtra)
# state directories
table.dir <- "tables"
table.file <- "HOBOreformat_BEFORE_20240226.xlsx"
table.sheet <- "Sheet1"
table <- read_excel(file.path(table.dir, table.file), sheet = table.sheet
, na = c("NA", ""), trim_ws = TRUE, skip = 0
, col_names = TRUE)
# Munge
## remove extra colnames
colnames(table)[2:6] <- ""
options(knitr.kable.NA = '')
# kable(table1)
table %>%
kbl() %>%
kable_styling(full_width = F, position = "left")
cat("\n\n")
```{r, fig.alt="HOBO U-20 file before running the reformat function"}
knitr::include_graphics("RMD_Images/HOBOreformat_BEFORE.jpg")
```

**After**

```{r}
library(readxl)
library(knitr)
library(kableExtra)
# state directories
table.dir <- "tables"
table.file <- "HOBOreformat_AFTER_20240226.xlsx"
table.sheet <- "Sheet1"
table <- read_excel(file.path(table.dir, table.file), sheet = table.sheet
, na = c("NA", ""), trim_ws = TRUE, skip = 0
, col_names = TRUE)
options(knitr.kable.NA = '')
# kable(table1)
table %>%
kbl() %>%
kable_styling(full_width = F, position = "left")
```{r, fig.alt="HOBO U-20 file after running the reformat function."}
knitr::include_graphics("RMD_Images/HOBOreformat_AFTER.jpg")
```
20 changes: 2 additions & 18 deletions inst/shiny-examples/ContDataQC/rmd/App_2a2_miniDOT1.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -33,24 +33,8 @@ When data are initially downloaded from miniDOT sensors, there are separate

Below is an example of what the output file looks like after running the **Concatenate** function.

```{r}
library(readxl)
library(knitr)
library(kableExtra)
# state directories
table.dir <- "tables"
table.file <- "miniDOT_After_Concatenate_20240226.xlsx"
table.sheet <- "Sheet1"
table <- read_excel(file.path(table.dir, table.file), sheet = table.sheet
, na = c("NA", ""), trim_ws = TRUE, skip = 0
, col_names = TRUE)
options(knitr.kable.NA = '')
# kable(table1)
table %>%
kbl() %>%
kable_styling(full_width = F, position = "left")
```{r, fig.alt="miniDOT DO file after running the Concatenate function."}
knitr::include_graphics("RMD_Images/miniDOT_AFTER_Concatenate.jpg")
```

## Additional Notes
Expand Down
43 changes: 5 additions & 38 deletions inst/shiny-examples/ContDataQC/rmd/App_2a3_miniDOT2.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -52,48 +52,15 @@ Below is an example of what files look like before and after running the **Refor

**Before**

```{r}
library(readxl)
library(knitr)
library(kableExtra)
# state directories
table.dir <- "tables"
table.file <- "miniDOTreformat_ BEFORE_20240226.xlsx"
table.sheet <- "Sheet1"
table <- read_excel(file.path(table.dir, table.file), sheet = table.sheet
, na = c("NA", ""), trim_ws = TRUE, skip = 0
, col_names = TRUE)
options(knitr.kable.NA = '')
# kable(table1)
table %>%
kbl() %>%
kable_styling(full_width = F, position = "left")
cat("\n\n")

```{r, fig.alt="miniDOT DO file before running the reformat function."}
knitr::include_graphics("RMD_Images/miniDOT_Before_Reformat.jpg")
```

**After**

```{r}
library(readxl)
library(knitr)
library(kableExtra)
# state directories
table.dir <- "tables"
table.file <- "miniDOTreformat_ AFTER_20240226.xlsx"
table.sheet <- "Sheet1"
table <- read_excel(file.path(table.dir, table.file), sheet = table.sheet
, na = c("NA", ""), trim_ws = TRUE, skip = 0
, col_names = TRUE)
options(knitr.kable.NA = '')
# kable(table1)
table %>%
kbl() %>%
kable_styling(full_width = F, position = "left")
```{r, fig.alt="miniDOT DO file after running the reformat function."}
knitr::include_graphics("RMD_Images/miniDOT_After_Reformat.jpg")
```

## Additional Notes
Expand Down
5 changes: 4 additions & 1 deletion inst/shiny-examples/ContDataQC/rmd/App_3c3_QCThresh_Eval.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ if(boo_DEBUG==TRUE){
If you have one or more years of continuous data for a site, we encourage you to evaluate the performance of the QC test thresholds for each parameter at that site and customize the configuration file if needed. Make sure you consider what units you are using, since units have a large effect on thresholds.

**Resources for evaluating thresholds:**

* [Pivot tables and charts](EvaluateThresholds.zip) [ZIP] for evaluating the Unrealistic values ('Gross range') and Spike tests in Excel.

* [R code statistics](TimMartin_R_ThresholdEval.zip) [ZIP] that can help inform thresholds for all four QC tests.
* [Threshold evaluation worksheet](ThresholdsCheckWorksheet_20220826.xlsx) [XLSX] that lists the default thresholds for each parameter and has columns where you enter customized thresholds for one or multiple sites.

* [Threshold evaluation worksheet](ThresholdsCheckWorksheet_20220826.xlsx) [XLSX] that lists the default thresholds for each parameter and has columns where you enter customized thresholds for one or multiple sites.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Binary file not shown.
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ <h2>miniDOT DO Sensors</h2>
<li><p><strong>miniDOT_reformat</strong> – run the combined file through
the Reformat function. The function reformats the file so that it is
ready to run through the QC report function. Next, go to the Main
<strong>Functions-Import Files</strong>s tab, import the reformatted
<strong>Functions-Import Files</strong> tab, import the reformatted
file, run it through the ‘QC raw data’ function, and check the flagged
data (edit if needed).</p></li>
</ul>
Expand Down
4 changes: 2 additions & 2 deletions inst/shiny-examples/ContDataQC/www/RMD_HTML/App_1c_FAQ.html
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ <h1>Basic Information</h1>
can run ContDataQC as a Shiny app on your local computer and it will
likely be faster. Running it locally requires that you have R software
and the ContDataQC R package installed on your computer, which can be
downloaded from GitHub
downloaded from
<a class="menu__link" href="https://github.com/leppott/ContDataQC" target="_blank">GitHub<span class="usa-tag external-link__tag" title="Exit EPA Website"> <span aria-hidden="true">Exit</span> <span class="u-visually-hidden"> Exit EPA
Website</span> </span> </a>.</p>
<p><strong>Internet browsers:</strong> This Shiny app has been tested
Expand Down Expand Up @@ -49,7 +49,7 @@ <h1>Basic Information</h1>
<p><strong>Data summary and visualization:</strong> The ContDataSumViz
app has more summary and visualization options. Shiny app (beta
version):
<a class="menu__link" href="https://contdataqcsumviz_containerized_stg.app.cloud.gov/" target="_blank"><a href="https://contdataqcsumviz_containerized_stg.app.cloud.gov/" class="uri">https://contdataqcsumviz_containerized_stg.app.cloud.gov/</a><span class="usa-tag external-link__tag" title="Exit EPA Website"> <span aria-hidden="true">Exit</span> <span class="u-visually-hidden"> Exit EPA
<a class="menu__link" href="https://dmap-contdataqcsumviz.app.cloud.gov/" target="_blank"><a href="https://dmap-contdataqcsumviz.app.cloud.gov/" class="uri">https://dmap-contdataqcsumviz.app.cloud.gov/</a><span class="usa-tag external-link__tag" title="Exit EPA Website"> <span aria-hidden="true">Exit</span> <span class="u-visually-hidden"> Exit EPA
Website</span> </span> </a></p>
<p>R code:
<a class="menu__link" href="https://github.com/USEPA/dmap-ContDataQCSumViz" target="_blank"><a href="https://github.com/USEPA/dmap-ContDataQCSumViz" class="uri">https://github.com/USEPA/dmap-ContDataQCSumViz</a><span class="usa-tag external-link__tag" title="Exit EPA Website"> <span aria-hidden="true">Exit</span> <span class="u-visually-hidden"> Exit EPA
Expand Down
Loading

0 comments on commit 96a36ae

Please sign in to comment.