v2.0.7.9015

* refactor: Shiny tab text updates + Check Input + Main + Organize Files + HOBOware Reformat * refactor: Rename Sensor File Modification to Automated Reformatting + Add About tab
leppott · Sep 8, 2023 · a86bceb · a86bceb
1 parent 6158f6d
commit a86bceb
Show file tree

Hide file tree

Showing 17 changed files with 197 additions and 83 deletions.
diff --git a/DESCRIPTION b/DESCRIPTION
@@ -1,6 +1,6 @@
 Package: ContDataQC
 Title: Quality Control (QC) of Continous Monitoring Data
-Version: 2.0.7.9014
+Version: 2.0.7.9015
 Authors@R: c(
 	person("Erik W", "Leppo", email="[email protected]",role=c("aut","cre")),
 	person("Ann","Roseberry Lincoln", role="ctb"),

diff --git a/NEWS b/NEWS
@@ -3,10 +3,22 @@ NEWS-ContDataQC
 
 <!-- NEWS.md is generated from NEWS.Rmd. Please edit that file -->
 
-    #> Last Update: 2023-09-08 14:58:39.04085
+    #> Last Update: 2023-09-08 15:57:43.083915
 
 # Version History
 
+## v2.0.7.9015
+
+2023-09-08
+
+- refactor: Shiny tab text updates
+  - Check Input
+  - Main
+  - Organize Files
+  - HOBOware Reformat
+- refactor: Rename Sensor File Modification to Automated Reformatting
+  - Add About tab
+
 ## v2.0.7.9014
 
 2023-09-08

diff --git a/NEWS.md b/NEWS.md
@@ -3,10 +3,22 @@ NEWS-ContDataQC
 
 <!-- NEWS.md is generated from NEWS.Rmd. Please edit that file -->
 
-    #> Last Update: 2023-09-08 14:58:39.04085
+    #> Last Update: 2023-09-08 15:57:43.083915
 
 # Version History
 
+## v2.0.7.9015
+
+2023-09-08
+
+- refactor: Shiny tab text updates
+  - Check Input
+  - Main
+  - Organize Files
+  - HOBOware Reformat
+- refactor: Rename Sensor File Modification to Automated Reformatting
+  - Add About tab
+
 ## v2.0.7.9014
 
 2023-09-08

diff --git a/NEWS.rmd b/NEWS.rmd
@@ -21,6 +21,17 @@ cat(paste0("Last Update: ",Sys.time()))
 
 # Version History
 
+## v2.0.7.9015
+2023-09-08
+
+* refactor: Shiny tab text updates
+    + Check Input
+    + Main
+    + Organize Files
+    + HOBOware Reformat
+* refactor: Rename Sensor File Modification to Automated Reformatting
+    + Add About tab
+
 ## v2.0.7.9014
 2023-09-08
 

diff --git a/inst/shiny-examples/ContDataQC/external/tab_2_DataPrep.R b/inst/shiny-examples/ContDataQC/external/tab_2_DataPrep.R
@@ -10,8 +10,11 @@ function() {
                                     , includeHTML("www/RMD_HTML/App_2a_FormattingFiles.html")
                          )## tabPanel ~ END
 
-                         , tabPanel("Sensor File Modification"
+                         , tabPanel("Automated Reformatting"
                             ,tabsetPanel(type = "tabs"
+                                         , tabPanel("About"
+                                                    , includeHTML("www/RMD_HTML/App_2az_About.html")
+                                         )## tabPanel ~ END
                                          # Hoboware, reformat ----
                                          , tabPanel("HOBOware Reformat"
                                                     ,sidebarLayout(

diff --git a/inst/shiny-examples/ContDataQC/rmd/App_2a1_HOBO.rmd b/inst/shiny-examples/ContDataQC/rmd/App_2a1_HOBO.rmd
@@ -11,15 +11,15 @@ output:
 knitr::opts_chunk$set(results='asis', echo=FALSE, warning=FALSE, message = FALSE)
 # needed for trouble shooting
 boo_DEBUG <- FALSE
-if(boo_DEBUG==TRUE){
+if(boo_DEBUG == TRUE){
   # myConfig <- file.path(system.file(package="ContDataQC"), "extdata", "config.ORIG.R")
   # source(myConfig)
 }## IF ~ boo_DEBUG ~ END
 ```
 
 # HOBOware Reformat
 
-The HOBOware reformat function is for data files from Onset HOBO sensors. It converts them to a format that is ready to run through the DataQC function. This saves you time and eliminates errors that sometimes occur when manually reformatting files.
+The HOBOware reformat function is for data files from Onset HOBO sensors. It converts them to a format that is ready to run through the QC report function (Main Functions - QC raw data). This saves you time and eliminates errors that sometimes occur when manually reformatting files.
 
 HOBO sensor files must first be opened with HOBOware software, then processed and exported as CSV files. If you follow these instructions, you will be able to run them through the reformat function without manipulation -
 
@@ -39,20 +39,17 @@ HOBO sensor files must first be opened with HOBOware software, then processed an
 
 *	Select the Date format in your input file (options: YMD, MDY, DMY)
 
-  +	Note: if you enter the wrong date format, the function will still run but 
-  your date field will be blank and you will not receive an error message. We 
-  recommend checking your output files to make sure the date field came through 
-  (as well as the other desired parameters; if those are not named correctly, 
-  they will be missing from the output file and you will not receive an error 
-  message).
-
 *	Click the ‘Reformat HOBOware file(s)’ button; a status bar will appear in the lower right corner.
 
 *	After the function is done running, a ‘Download’ button will appear. Click this button.
 
-*	Save the zipped file in the Data1_RAW folder
+* You will be prompted to save a zipped file named ‘format_HOBO_date_time’. Save the zipped file in the Data1_RAW folder.
+
+*	Unzip/extract the file(s) or else the files won’t be seen as an option to upload to the QC report function (Main Functions - QC raw data). After file is unzipped, a new folder called ‘HOBO’ will appear. The original zipped file can be deleted if desired.
 
-*	Time-saver tip: if you use Google Chrome as your internet browser, you can change its settings so that it asks what folder to download files to (versus files going to the default Download folder). Click [here](GoogleChrome_TimesaverTips.pdf) for instructions on how to set this up. Your files are now ready to be run through the QC function!
+Time-saver tip: some internet browsers, like Google Chrome and Mozilla Firefox, allow users to change settings so that users can select which folder to download files to (versus files going to the default Download folder). Click [here](GoogleChrome_TimesaverTips.pdf) for instructions on how to set this up in Chrome. With Mozilla Firefox, go to Settings > General > Files and Applications > Check box for “always ask where to save files.” 
+
+Your files are now ready to be run through the QC function!
 Below are examples of what HOBO U20 CSV files look like before and after they are run through the reformat function. The reformat function does the following -
 
 *	Changes column headings as needed
@@ -69,6 +66,13 @@ If there is a problem with the input file, the screen will turn gray and it will
 
 ### Common errors
 
+* If you enter the wrong date format, the function will still run but 
+your date field will be blank and you will not receive an error message. We 
+recommend checking your output files to make sure the date field came through 
+(as well as the other desired parameters; if those are not named correctly, 
+they will be missing from the output file and you will not receive an error 
+message).
+
 *	Unwanted ‘logged’ text entries in the bottom row of the CSV. These occur if you forget to uncheck the ‘select internal logger events to plot’ boxes during Plot SetUp in HOBOware
 
 *	Two serial numbers in the column heading (you should only have one S/N; make sure under Preferences-Export settings, you only have the ‘include logger serial number’ box checked, NOT the ‘include sensor serial number or label if available’ box)

diff --git a/inst/shiny-examples/ContDataQC/rmd/App_2az_About.rmd b/inst/shiny-examples/ContDataQC/rmd/App_2az_About.rmd
@@ -0,0 +1,32 @@
+---
+title: "Automated Reformating, About"
+date: "`r format(Sys.time(), '%Y-%m-%d %H:%M:%S')`"
+always_allow_html: true
+output:
+  html_fragment:
+    toc: yes
+---
+
+```{r setup, include=FALSE}
+knitr::opts_chunk$set(results='asis', echo=FALSE, warning=FALSE, message = FALSE)
+# needed for trouble shooting
+boo_DEBUG <- FALSE
+if(boo_DEBUG==TRUE){
+  # myConfig <- file.path(system.file(package="ContDataQC"), "extdata", "config.ORIG.R")
+  # source(myConfig)
+}## IF ~ boo_DEBUG ~ END
+```
+
+# Automated Reformatting
+
+These functions convert files coming directly from sensors or proprietary sensor software like HOBOware into a format that is ready to run through the QC report function (**Main Functions - QC raw data**). This saves you time and eliminates errors that sometimes occur when manually reformatting files.
+
+Currently, reformatting functions are available for two types of sensors:
+
+*	Onset HOBO (water level (U20), temperature (pro V2 and Pendant), and DO (U26); more can be added as needed)
+
+*	miniDOT DO
+
+Reformatting functions for other types of sensors can be added in the future, if requested.
+
+
diff --git a/inst/shiny-examples/ContDataQC/rmd/App_2b_OrganizingFiles.rmd b/inst/shiny-examples/ContDataQC/rmd/App_2b_OrganizingFiles.rmd
@@ -8,10 +8,10 @@ output:
 ---
 
 ```{r setup, include=FALSE}
-knitr::opts_chunk$set(results='asis', echo=FALSE, warning=FALSE, message = FALSE)
+knitr::opts_chunk$set(results = 'asis', echo = FALSE, warning = FALSE, message = FALSE)
 # needed for trouble shooting
 boo_DEBUG <- FALSE
-if(boo_DEBUG==TRUE){
+if (boo_DEBUG == TRUE) {
   # myConfig <- file.path(system.file(package="ContDataQC"), "extdata", "config.ORIG.R")
   # source(myConfig)
 }## IF ~ boo_DEBUG ~ END
@@ -24,7 +24,7 @@ we recommend setting up folders as shown below as a starting point. Folders and
 files can be customized as you gain more experience. The files should be backed
 up regularly and accessible to other members of your data management team.
 
-Click [here](FolderSetUp_20220825.zip) to download a zipped file with a 
+Click [here](SiteX_20230828.zip) to download a zipped file with a 
 ready-made set of (empty) folders.  Put the file in the desired location, unzip 
 it and then replace ‘SiteX’ with the appropriate site name.
 

diff --git a/inst/shiny-examples/ContDataQC/rmd/App_3a1_About.rmd b/inst/shiny-examples/ContDataQC/rmd/App_3a1_About.rmd
@@ -17,29 +17,21 @@ if(boo_DEBUG==TRUE){
 }## IF ~ boo_DEBUG ~ END
 ```
 
-# Run Functions
+# Main Functions
 
-The three main functions are listed in the drop-down menu under 'Choose 
-Operation to perform'
+_Workflow_
 
--   **'QC raw data'** generates QC reports.
+1.	**Select QC flag test thresholds.** Use the default thresholds or upload a customized configuration file. If you have one or more years of continuous data for a site, we encourage you to evaluate the performance of the QC test thresholds for each parameter at that site and customize the configuration file if needed. Click here for tips on evaluating thresholds. To learn how to edit thresholds, click here.
 
--   **'Aggregate QC'd data'** merges the QC'd data files for the desired time 
-periods together. The input files need to be from the same site and share common
-parameters.
+2.	**Check the configuration file name.** Make sure the correct configuration file has been uploaded prior to running the QC reports.
 
--   **'Summary statistics'** generates two sets of CSV files and a PDF with time
-series plots for each parameter. One of the CSVs has daily means and the other 
-has a wider range of summary statistics (e.g., min, median, max, quartiles).
+3.	**Upload the desired input file(s).**
 
-ContDataQC works sequentially, meaning the user needs to run the QC function 
-before running the Aggregate function, and the Aggregate function before the 
-SummaryStats function. More detailed information on each function can be found 
-below.
+4.	**Run the ‘QC raw data’** function to generate QC reports.
 
-Time-saver tip: if you use Google Chrome as your internet browser, you can 
-change its settings so that it asks what folder to download files to (versus 
-files going to the default Download folder). Click 
-[here](GoogleChrome_TimesaverTips.pdf) for instructions on how to set this up.
+5.	**After the data file(s) are QC’d, users can run them through the Aggregate function** to merge them with other files. The input files need to be from the same site and share common parameters.
 
+6.	**The user can run the aggregated file through the Summary Statistics function** to generate CSV files with an assortment of summary statistics (e.g., min, median, max, quartiles) and a PDF with time series plots for each parameter. 
+After running each function, users must unzip/extract the files or else the files won’t be seen as an option to upload for the next step. After files are unzipped, new folders will automatically appear and the original zipped files can be deleted if desired.
+Time-saver tip: some internet browsers, like Google Chrome and Mozilla Firefox, allow users to change settings so that users can select which folder to download files to (versus files going to the default Download folder). Click [here](GoogleChrome_TimesaverTips.pdf) for instructions on how to set this up in Chrome. With Mozilla Firefox, go to Settings > General > Files and Applications > Check box for “always ask where to save files.”
 
diff --git a/inst/shiny-examples/ContDataQC/rmd/tables/App_Information_Tables.xlsx b/inst/shiny-examples/ContDataQC/rmd/tables/App_Information_Tables.xlsx
diff --git a/inst/shiny-examples/ContDataQC/server.R b/inst/shiny-examples/ContDataQC/server.R
@@ -241,12 +241,13 @@ shinyServer(function(input, output, session) {
   #Creates a summary data.frame as a reactive object.
   #This table includes file name, station ID, start date, end date, and record count.
   table1 <- reactive({
+    # 20230908, drop dates, cols 3 and 4
 
     #Shows the table headings before files are input
     if (is.null(allFiles())) {
 
       #Subsets the columns of the pre-upload data for display
-      nullTable1 <- fileAttribsNull()[c(1:5)]
+      nullTable1 <- fileAttribsNull()[c(1, 2, 5)]
 
       #Sends the empty table to be displayed
       return(nullTable1)
@@ -255,7 +256,7 @@ shinyServer(function(input, output, session) {
     #Subsets the file attribute table with just
     #file name, site ID, start date, end date, and number of records
     summaryTable1 <- fileAttribsFull()[, c(1:5)]
-    colnames(summaryTable1) <- colnames(fileAttribsFull()[c(1:5)])
+    colnames(summaryTable1) <- colnames(fileAttribsFull()[c(1, 2, 5)])
 
     return(summaryTable1)
   })

diff --git a/inst/shiny-examples/ContDataQC/ui.R b/inst/shiny-examples/ContDataQC/ui.R
@@ -25,7 +25,7 @@ tab_5b_Console    <- source("external/tab_5b_Console.R", local = TRUE)$value
 
 shinyUI(
   # VERSION, 1, current [non-EPA] ----
-  navbarPage("Continuous data QC, summary, and statistics - v2.0.7.9014",
+  navbarPage("Continuous data QC, summary, and statistics - v2.0.7.9015",
              theme = shinytheme("spacelab")
              ,tab_1_About()
              #,tab_1a_Overview()

diff --git a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2a1_HOBO.html b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2a1_HOBO.html
@@ -5,8 +5,9 @@
 <h1>HOBOware Reformat</h1>
 <p>The HOBOware reformat function is for data files from Onset HOBO
 sensors. It converts them to a format that is ready to run through the
-DataQC function. This saves you time and eliminates errors that
-sometimes occur when manually reformatting files.</p>
+QC report function (Main Functions - QC raw data). This saves you time
+and eliminates errors that sometimes occur when manually reformatting
+files.</p>
 <p>HOBO sensor files must first be opened with HOBOware software, then
 processed and exported as CSV files. If you follow these instructions,
 you will be able to run them through the reformat function without
@@ -31,25 +32,30 @@ <h3>Instructions</h3>
 is complete.</p></li>
 <li><p>Select the Date format in your input file (options: YMD, MDY,
 DMY)</p></li>
-<li><p>Note: if you enter the wrong date format, the function will still
-run but your date field will be blank and you will not receive an error
-message. We recommend checking your output files to make sure the date
-field came through (as well as the other desired parameters; if those
-are not named correctly, they will be missing from the output file and
-you will not receive an error message).</p></li>
 <li><p>Click the ‘Reformat HOBOware file(s)’ button; a status bar will
 appear in the lower right corner.</p></li>
 <li><p>After the function is done running, a ‘Download’ button will
 appear. Click this button.</p></li>
-<li><p>Save the zipped file in the Data1_RAW folder</p></li>
-<li><p>Time-saver tip: if you use Google Chrome as your internet
-browser, you can change its settings so that it asks what folder to
-download files to (versus files going to the default Download folder).
-Click <a href="GoogleChrome_TimesaverTips.pdf">here</a> for instructions
-on how to set this up. Your files are now ready to be run through the QC
-function! Below are examples of what HOBO U20 CSV files look like before
-and after they are run through the reformat function. The reformat
-function does the following -</p></li>
+<li><p>You will be prompted to save a zipped file named
+‘format_HOBO_date_time’. Save the zipped file in the Data1_RAW
+folder.</p></li>
+<li><p>Unzip/extract the file(s) or else the files won’t be seen as an
+option to upload to the QC report function (Main Functions - QC raw
+data). After file is unzipped, a new folder called ‘HOBO’ will appear.
+The original zipped file can be deleted if desired.</p></li>
+</ul>
+<p>Time-saver tip: some internet browsers, like Google Chrome and
+Mozilla Firefox, allow users to change settings so that users can select
+which folder to download files to (versus files going to the default
+Download folder). Click <a href="GoogleChrome_TimesaverTips.pdf">here</a> for instructions on how
+to set this up in Chrome. With Mozilla Firefox, go to Settings &gt;
+General &gt; Files and Applications &gt; Check box for “always ask where
+to save files.”</p>
+<p>Your files are now ready to be run through the QC function! Below are
+examples of what HOBO U20 CSV files look like before and after they are
+run through the reformat function. The reformat function does the
+following -</p>
+<ul>
 <li><p>Changes column headings as needed</p></li>
 <li><p>Creates a new ‘SiteID’ column and populates it with the SiteID
 that it takes from the first part of the input file name (the SiteID
@@ -65,6 +71,12 @@ <h3>Instructions</h3>
 <div id="common-errors" class="section level3">
 <h3>Common errors</h3>
 <ul>
+<li><p>If you enter the wrong date format, the function will still run
+but your date field will be blank and you will not receive an error
+message. We recommend checking your output files to make sure the date
+field came through (as well as the other desired parameters; if those
+are not named correctly, they will be missing from the output file and
+you will not receive an error message).</p></li>
 <li><p>Unwanted ‘logged’ text entries in the bottom row of the CSV.
 These occur if you forget to uncheck the ‘select internal logger events
 to plot’ boxes during Plot SetUp in HOBOware</p></li>

diff --git a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2az_About.html b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2az_About.html
@@ -0,0 +1,20 @@
+
+
+
+<div id="automated-reformatting" class="section level1">
+<h1>Automated Reformatting</h1>
+<p>These functions convert files coming directly from sensors or
+proprietary sensor software like HOBOware into a format that is ready to
+run through the QC report function (<strong>Main Functions - QC raw
+data</strong>). This saves you time and eliminates errors that sometimes
+occur when manually reformatting files.</p>
+<p>Currently, reformatting functions are available for two types of
+sensors:</p>
+<ul>
+<li><p>Onset HOBO (water level (U20), temperature (pro V2 and Pendant),
+and DO (U26); more can be added as needed)</p></li>
+<li><p>miniDOT DO</p></li>
+</ul>
+<p>Reformatting functions for other types of sensors can be added in the
+future, if requested.</p>
+</div>
diff --git a/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2b_OrganizingFiles.html b/inst/shiny-examples/ContDataQC/www/RMD_HTML/App_2b_OrganizingFiles.html
@@ -8,10 +8,10 @@ <h1>Organizing Files</h1>
 point. Folders and files can be customized as you gain more experience.
 The files should be backed up regularly and accessible to other members
 of your data management team.</p>
-<p>Click <a href="FolderSetUp_20220825.zip">here</a> to download a
-zipped file with a ready-made set of (empty) folders. Put the file in
-the desired location, unzip it and then replace ‘SiteX’ with the
-appropriate site name.</p>
+<p>Click <a href="SiteX_20230828.zip">here</a> to download a zipped file
+with a ready-made set of (empty) folders. Put the file in the desired
+location, unzip it and then replace ‘SiteX’ with the appropriate site
+name.</p>
 <p>You have more flexibility with file organization when using the Shiny
 app vs R package. With the Shiny app, it is not necessary to name the
 folders exactly as shown below because you can browse to the desired
@@ -168,7 +168,8 @@ <h3>Folder descriptions</h3>
 <td style="text-align:left;">
 If there is a nearby, representative weather station, we recommend
 comparing your sensor data to the weather station data as an additional
-QC check.
+QC check. Modeled daily air temperature and precipitation data are also
+available from Daymet (<a href="https://daymet.ornl.gov/" class="uri">https://daymet.ornl.gov/</a>) and PRISM (<a href="https://prism.oregonstate.edu/" class="uri">https://prism.oregonstate.edu/</a>).
 </td>
 </tr>
 </tbody>