Skip to content

Commit

Permalink
v2.0.7.9015
Browse files Browse the repository at this point in the history
* refactor: Shiny tab text updates
    + Check Input
    + Main
    + Organize Files
    + HOBOware Reformat
* refactor: Rename Sensor File Modification to Automated Reformatting
    + Add About tab
  • Loading branch information
leppott committed Sep 8, 2023
1 parent 6158f6d commit a86bceb
Show file tree
Hide file tree
Showing 17 changed files with 197 additions and 83 deletions.
2 changes: 1 addition & 1 deletion DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Package: ContDataQC
Title: Quality Control (QC) of Continous Monitoring Data
Version: 2.0.7.9014
Version: 2.0.7.9015
Authors@R: c(
person("Erik W", "Leppo", email="[email protected]",role=c("aut","cre")),
person("Ann","Roseberry Lincoln", role="ctb"),
Expand Down
14 changes: 13 additions & 1 deletion NEWS
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,22 @@ NEWS-ContDataQC

<!-- NEWS.md is generated from NEWS.Rmd. Please edit that file -->

#> Last Update: 2023-09-08 14:58:39.04085
#> Last Update: 2023-09-08 15:57:43.083915

# Version History

## v2.0.7.9015

2023-09-08

- refactor: Shiny tab text updates
- Check Input
- Main
- Organize Files
- HOBOware Reformat
- refactor: Rename Sensor File Modification to Automated Reformatting
- Add About tab

## v2.0.7.9014

2023-09-08
Expand Down
14 changes: 13 additions & 1 deletion NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,22 @@ NEWS-ContDataQC

<!-- NEWS.md is generated from NEWS.Rmd. Please edit that file -->

#> Last Update: 2023-09-08 14:58:39.04085
#> Last Update: 2023-09-08 15:57:43.083915

# Version History

## v2.0.7.9015

2023-09-08

- refactor: Shiny tab text updates
- Check Input
- Main
- Organize Files
- HOBOware Reformat
- refactor: Rename Sensor File Modification to Automated Reformatting
- Add About tab

## v2.0.7.9014

2023-09-08
Expand Down
11 changes: 11 additions & 0 deletions NEWS.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,17 @@ cat(paste0("Last Update: ",Sys.time()))

# Version History

## v2.0.7.9015
2023-09-08

* refactor: Shiny tab text updates
+ Check Input
+ Main
+ Organize Files
+ HOBOware Reformat
* refactor: Rename Sensor File Modification to Automated Reformatting
+ Add About tab

## v2.0.7.9014
2023-09-08

Expand Down
5 changes: 4 additions & 1 deletion inst/shiny-examples/ContDataQC/external/tab_2_DataPrep.R
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,11 @@ function() {
, includeHTML("www/RMD_HTML/App_2a_FormattingFiles.html")
)## tabPanel ~ END

, tabPanel("Sensor File Modification"
, tabPanel("Automated Reformatting"
,tabsetPanel(type = "tabs"
, tabPanel("About"
, includeHTML("www/RMD_HTML/App_2az_About.html")
)## tabPanel ~ END
# Hoboware, reformat ----
, tabPanel("HOBOware Reformat"
,sidebarLayout(
Expand Down
26 changes: 15 additions & 11 deletions inst/shiny-examples/ContDataQC/rmd/App_2a1_HOBO.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,15 @@ output:
knitr::opts_chunk$set(results='asis', echo=FALSE, warning=FALSE, message = FALSE)
# needed for trouble shooting
boo_DEBUG <- FALSE
if(boo_DEBUG==TRUE){
if(boo_DEBUG == TRUE){
# myConfig <- file.path(system.file(package="ContDataQC"), "extdata", "config.ORIG.R")
# source(myConfig)
}## IF ~ boo_DEBUG ~ END
```

# HOBOware Reformat

The HOBOware reformat function is for data files from Onset HOBO sensors. It converts them to a format that is ready to run through the DataQC function. This saves you time and eliminates errors that sometimes occur when manually reformatting files.
The HOBOware reformat function is for data files from Onset HOBO sensors. It converts them to a format that is ready to run through the QC report function (Main Functions - QC raw data). This saves you time and eliminates errors that sometimes occur when manually reformatting files.

HOBO sensor files must first be opened with HOBOware software, then processed and exported as CSV files. If you follow these instructions, you will be able to run them through the reformat function without manipulation -

Expand All @@ -39,20 +39,17 @@ HOBO sensor files must first be opened with HOBOware software, then processed an

* Select the Date format in your input file (options: YMD, MDY, DMY)

+ Note: if you enter the wrong date format, the function will still run but
your date field will be blank and you will not receive an error message. We
recommend checking your output files to make sure the date field came through
(as well as the other desired parameters; if those are not named correctly,
they will be missing from the output file and you will not receive an error
message).

* Click the ‘Reformat HOBOware file(s)’ button; a status bar will appear in the lower right corner.

* After the function is done running, a ‘Download’ button will appear. Click this button.

* Save the zipped file in the Data1_RAW folder
* You will be prompted to save a zipped file named ‘format_HOBO_date_time’. Save the zipped file in the Data1_RAW folder.

* Unzip/extract the file(s) or else the files won’t be seen as an option to upload to the QC report function (Main Functions - QC raw data). After file is unzipped, a new folder called ‘HOBO’ will appear. The original zipped file can be deleted if desired.

* Time-saver tip: if you use Google Chrome as your internet browser, you can change its settings so that it asks what folder to download files to (versus files going to the default Download folder). Click [here](GoogleChrome_TimesaverTips.pdf) for instructions on how to set this up. Your files are now ready to be run through the QC function!
Time-saver tip: some internet browsers, like Google Chrome and Mozilla Firefox, allow users to change settings so that users can select which folder to download files to (versus files going to the default Download folder). Click [here](GoogleChrome_TimesaverTips.pdf) for instructions on how to set this up in Chrome. With Mozilla Firefox, go to Settings > General > Files and Applications > Check box for “always ask where to save files.”

Your files are now ready to be run through the QC function!
Below are examples of what HOBO U20 CSV files look like before and after they are run through the reformat function. The reformat function does the following -

* Changes column headings as needed
Expand All @@ -69,6 +66,13 @@ If there is a problem with the input file, the screen will turn gray and it will

### Common errors

* If you enter the wrong date format, the function will still run but
your date field will be blank and you will not receive an error message. We
recommend checking your output files to make sure the date field came through
(as well as the other desired parameters; if those are not named correctly,
they will be missing from the output file and you will not receive an error
message).

* Unwanted ‘logged’ text entries in the bottom row of the CSV. These occur if you forget to uncheck the ‘select internal logger events to plot’ boxes during Plot SetUp in HOBOware

* Two serial numbers in the column heading (you should only have one S/N; make sure under Preferences-Export settings, you only have the ‘include logger serial number’ box checked, NOT the ‘include sensor serial number or label if available’ box)
Expand Down
32 changes: 32 additions & 0 deletions inst/shiny-examples/ContDataQC/rmd/App_2az_About.rmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
---
title: "Automated Reformating, About"
date: "`r format(Sys.time(), '%Y-%m-%d %H:%M:%S')`"
always_allow_html: true
output:
html_fragment:
toc: yes
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(results='asis', echo=FALSE, warning=FALSE, message = FALSE)
# needed for trouble shooting
boo_DEBUG <- FALSE
if(boo_DEBUG==TRUE){
# myConfig <- file.path(system.file(package="ContDataQC"), "extdata", "config.ORIG.R")
# source(myConfig)
}## IF ~ boo_DEBUG ~ END
```

# Automated Reformatting

These functions convert files coming directly from sensors or proprietary sensor software like HOBOware into a format that is ready to run through the QC report function (**Main Functions - QC raw data**). This saves you time and eliminates errors that sometimes occur when manually reformatting files.

Currently, reformatting functions are available for two types of sensors:

* Onset HOBO (water level (U20), temperature (pro V2 and Pendant), and DO (U26); more can be added as needed)

* miniDOT DO

Reformatting functions for other types of sensors can be added in the future, if requested.


6 changes: 3 additions & 3 deletions inst/shiny-examples/ContDataQC/rmd/App_2b_OrganizingFiles.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ output:
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(results='asis', echo=FALSE, warning=FALSE, message = FALSE)
knitr::opts_chunk$set(results = 'asis', echo = FALSE, warning = FALSE, message = FALSE)
# needed for trouble shooting
boo_DEBUG <- FALSE
if(boo_DEBUG==TRUE){
if (boo_DEBUG == TRUE) {
# myConfig <- file.path(system.file(package="ContDataQC"), "extdata", "config.ORIG.R")
# source(myConfig)
}## IF ~ boo_DEBUG ~ END
Expand All @@ -24,7 +24,7 @@ we recommend setting up folders as shown below as a starting point. Folders and
files can be customized as you gain more experience. The files should be backed
up regularly and accessible to other members of your data management team.

Click [here](FolderSetUp_20220825.zip) to download a zipped file with a
Click [here](SiteX_20230828.zip) to download a zipped file with a
ready-made set of (empty) folders. Put the file in the desired location, unzip
it and then replace ‘SiteX’ with the appropriate site name.

Expand Down
28 changes: 10 additions & 18 deletions inst/shiny-examples/ContDataQC/rmd/App_3a1_About.rmd
Original file line number Diff line number Diff line change
Expand Up @@ -17,29 +17,21 @@ if(boo_DEBUG==TRUE){
}## IF ~ boo_DEBUG ~ END
```

# Run Functions
# Main Functions

The three main functions are listed in the drop-down menu under 'Choose
Operation to perform'
_Workflow_

- **'QC raw data'** generates QC reports.
1. **Select QC flag test thresholds.** Use the default thresholds or upload a customized configuration file. If you have one or more years of continuous data for a site, we encourage you to evaluate the performance of the QC test thresholds for each parameter at that site and customize the configuration file if needed. Click here for tips on evaluating thresholds. To learn how to edit thresholds, click here.

- **'Aggregate QC'd data'** merges the QC'd data files for the desired time
periods together. The input files need to be from the same site and share common
parameters.
2. **Check the configuration file name.** Make sure the correct configuration file has been uploaded prior to running the QC reports.

- **'Summary statistics'** generates two sets of CSV files and a PDF with time
series plots for each parameter. One of the CSVs has daily means and the other
has a wider range of summary statistics (e.g., min, median, max, quartiles).
3. **Upload the desired input file(s).**

ContDataQC works sequentially, meaning the user needs to run the QC function
before running the Aggregate function, and the Aggregate function before the
SummaryStats function. More detailed information on each function can be found
below.
4. **Run the ‘QC raw data’** function to generate QC reports.

Time-saver tip: if you use Google Chrome as your internet browser, you can
change its settings so that it asks what folder to download files to (versus
files going to the default Download folder). Click
[here](GoogleChrome_TimesaverTips.pdf) for instructions on how to set this up.
5. **After the data file(s) are QC’d, users can run them through the Aggregate function** to merge them with other files. The input files need to be from the same site and share common parameters.

6. **The user can run the aggregated file through the Summary Statistics function** to generate CSV files with an assortment of summary statistics (e.g., min, median, max, quartiles) and a PDF with time series plots for each parameter.
After running each function, users must unzip/extract the files or else the files won’t be seen as an option to upload for the next step. After files are unzipped, new folders will automatically appear and the original zipped files can be deleted if desired.
Time-saver tip: some internet browsers, like Google Chrome and Mozilla Firefox, allow users to change settings so that users can select which folder to download files to (versus files going to the default Download folder). Click [here](GoogleChrome_TimesaverTips.pdf) for instructions on how to set this up in Chrome. With Mozilla Firefox, go to Settings > General > Files and Applications > Check box for “always ask where to save files.”

Binary file not shown.
5 changes: 3 additions & 2 deletions inst/shiny-examples/ContDataQC/server.R
Original file line number Diff line number Diff line change
Expand Up @@ -241,12 +241,13 @@ shinyServer(function(input, output, session) {
#Creates a summary data.frame as a reactive object.
#This table includes file name, station ID, start date, end date, and record count.
table1 <- reactive({
# 20230908, drop dates, cols 3 and 4

#Shows the table headings before files are input
if (is.null(allFiles())) {

#Subsets the columns of the pre-upload data for display
nullTable1 <- fileAttribsNull()[c(1:5)]
nullTable1 <- fileAttribsNull()[c(1, 2, 5)]

#Sends the empty table to be displayed
return(nullTable1)
Expand All @@ -255,7 +256,7 @@ shinyServer(function(input, output, session) {
#Subsets the file attribute table with just
#file name, site ID, start date, end date, and number of records
summaryTable1 <- fileAttribsFull()[, c(1:5)]
colnames(summaryTable1) <- colnames(fileAttribsFull()[c(1:5)])
colnames(summaryTable1) <- colnames(fileAttribsFull()[c(1, 2, 5)])

return(summaryTable1)
})
Expand Down
2 changes: 1 addition & 1 deletion inst/shiny-examples/ContDataQC/ui.R
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ tab_5b_Console <- source("external/tab_5b_Console.R", local = TRUE)$value

shinyUI(
# VERSION, 1, current [non-EPA] ----
navbarPage("Continuous data QC, summary, and statistics - v2.0.7.9014",
navbarPage("Continuous data QC, summary, and statistics - v2.0.7.9015",
theme = shinytheme("spacelab")
,tab_1_About()
#,tab_1a_Overview()
Expand Down
46 changes: 29 additions & 17 deletions inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2a1_HOBO.html
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@
<h1>HOBOware Reformat</h1>
<p>The HOBOware reformat function is for data files from Onset HOBO
sensors. It converts them to a format that is ready to run through the
DataQC function. This saves you time and eliminates errors that
sometimes occur when manually reformatting files.</p>
QC report function (Main Functions - QC raw data). This saves you time
and eliminates errors that sometimes occur when manually reformatting
files.</p>
<p>HOBO sensor files must first be opened with HOBOware software, then
processed and exported as CSV files. If you follow these instructions,
you will be able to run them through the reformat function without
Expand All @@ -31,25 +32,30 @@ <h3>Instructions</h3>
is complete.</p></li>
<li><p>Select the Date format in your input file (options: YMD, MDY,
DMY)</p></li>
<li><p>Note: if you enter the wrong date format, the function will still
run but your date field will be blank and you will not receive an error
message. We recommend checking your output files to make sure the date
field came through (as well as the other desired parameters; if those
are not named correctly, they will be missing from the output file and
you will not receive an error message).</p></li>
<li><p>Click the ‘Reformat HOBOware file(s)’ button; a status bar will
appear in the lower right corner.</p></li>
<li><p>After the function is done running, a ‘Download’ button will
appear. Click this button.</p></li>
<li><p>Save the zipped file in the Data1_RAW folder</p></li>
<li><p>Time-saver tip: if you use Google Chrome as your internet
browser, you can change its settings so that it asks what folder to
download files to (versus files going to the default Download folder).
Click <a href="GoogleChrome_TimesaverTips.pdf">here</a> for instructions
on how to set this up. Your files are now ready to be run through the QC
function! Below are examples of what HOBO U20 CSV files look like before
and after they are run through the reformat function. The reformat
function does the following -</p></li>
<li><p>You will be prompted to save a zipped file named
‘format_HOBO_date_time’. Save the zipped file in the Data1_RAW
folder.</p></li>
<li><p>Unzip/extract the file(s) or else the files won’t be seen as an
option to upload to the QC report function (Main Functions - QC raw
data). After file is unzipped, a new folder called ‘HOBO’ will appear.
The original zipped file can be deleted if desired.</p></li>
</ul>
<p>Time-saver tip: some internet browsers, like Google Chrome and
Mozilla Firefox, allow users to change settings so that users can select
which folder to download files to (versus files going to the default
Download folder). Click <a href="GoogleChrome_TimesaverTips.pdf">here</a> for instructions on how
to set this up in Chrome. With Mozilla Firefox, go to Settings &gt;
General &gt; Files and Applications &gt; Check box for “always ask where
to save files.”</p>
<p>Your files are now ready to be run through the QC function! Below are
examples of what HOBO U20 CSV files look like before and after they are
run through the reformat function. The reformat function does the
following -</p>
<ul>
<li><p>Changes column headings as needed</p></li>
<li><p>Creates a new ‘SiteID’ column and populates it with the SiteID
that it takes from the first part of the input file name (the SiteID
Expand All @@ -65,6 +71,12 @@ <h3>Instructions</h3>
<div id="common-errors" class="section level3">
<h3>Common errors</h3>
<ul>
<li><p>If you enter the wrong date format, the function will still run
but your date field will be blank and you will not receive an error
message. We recommend checking your output files to make sure the date
field came through (as well as the other desired parameters; if those
are not named correctly, they will be missing from the output file and
you will not receive an error message).</p></li>
<li><p>Unwanted ‘logged’ text entries in the bottom row of the CSV.
These occur if you forget to uncheck the ‘select internal logger events
to plot’ boxes during Plot SetUp in HOBOware</p></li>
Expand Down
20 changes: 20 additions & 0 deletions inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2az_About.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@



<div id="automated-reformatting" class="section level1">
<h1>Automated Reformatting</h1>
<p>These functions convert files coming directly from sensors or
proprietary sensor software like HOBOware into a format that is ready to
run through the QC report function (<strong>Main Functions - QC raw
data</strong>). This saves you time and eliminates errors that sometimes
occur when manually reformatting files.</p>
<p>Currently, reformatting functions are available for two types of
sensors:</p>
<ul>
<li><p>Onset HOBO (water level (U20), temperature (pro V2 and Pendant),
and DO (U26); more can be added as needed)</p></li>
<li><p>miniDOT DO</p></li>
</ul>
<p>Reformatting functions for other types of sensors can be added in the
future, if requested.</p>
</div>
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ <h1>Organizing Files</h1>
point. Folders and files can be customized as you gain more experience.
The files should be backed up regularly and accessible to other members
of your data management team.</p>
<p>Click <a href="FolderSetUp_20220825.zip">here</a> to download a
zipped file with a ready-made set of (empty) folders. Put the file in
the desired location, unzip it and then replace ‘SiteX’ with the
appropriate site name.</p>
<p>Click <a href="SiteX_20230828.zip">here</a> to download a zipped file
with a ready-made set of (empty) folders. Put the file in the desired
location, unzip it and then replace ‘SiteX’ with the appropriate site
name.</p>
<p>You have more flexibility with file organization when using the Shiny
app vs R package. With the Shiny app, it is not necessary to name the
folders exactly as shown below because you can browse to the desired
Expand Down Expand Up @@ -168,7 +168,8 @@ <h3>Folder descriptions</h3>
<td style="text-align:left;">
If there is a nearby, representative weather station, we recommend
comparing your sensor data to the weather station data as an additional
QC check.
QC check. Modeled daily air temperature and precipitation data are also
available from Daymet (<a href="https://daymet.ornl.gov/" class="uri">https://daymet.ornl.gov/</a>) and PRISM (<a href="https://prism.oregonstate.edu/" class="uri">https://prism.oregonstate.edu/</a>).
</td>
</tr>
</tbody>
Expand Down
Loading

0 comments on commit a86bceb

Please sign in to comment.