importRecords for fields that don't exist should error

[spgarbet]

A user has requested the following:

For importRecords(), I’d like to have an error (vs. warning) for trying to upload fields that do not exist in the project. If that’s happening, I’ve done something I didn’t mean to do, so would like a hard stop.

[berryld1]

Or, not necessarily should error (leave the shoulds for you guys to decide!), but could we have the option to error, i.e., optional argument to set to return an error instead of warning, whose default could be to warn to match current behavior, if that's best? thank you!

[spgarbet]

If it's to be an argument, I would say we should make it fully configurable: nothing, warning, error. I would say the default should be error. I'm also open to it being a warning by default. How to specify? @couthcommander do you know of any examples in base that have a parameter that specifies how an issue is dealt with that we could follow?

[couthcommander]

I do not know of an example off hand. Something like this would handle all warnings within a function, but that might not be the desired effect.

fun <- function(warnings = c('warning','error','ignore')) {
  warningHandler <- c(0,2,-1)[match(match.arg(warnings), c('warning','error','ignore'))]
  op <- options(warn = warningHandler)
  on.exit(options(op))
  warning('I tried to warn you')
}

[spgarbet]

> ignore <- function(...) {}
> 
> packageRoutine <- function(someConditionFUN = warning)
+ {
+     # do work here till condition happens
+     someConditionFUN("Condition happened!")
+     # finish work here
+     "work"
+ }
> 
> packageRoutine(ignore)
[1] "work"
> packageRoutine(warning)
[1] "work"
Warning message:
In packageRoutine(warning) : Condition happened!
> packageRoutine(stop)
Error in packageRoutine(stop) : Condition happened!
> 

Is what I'm thinking. That way error handling is entirely defined by the user.

importRecords <- function(args, nonexistantFieldsFUN=stop, ...) { ... }

[spgarbet]

An interface design choice like this is part of #113

[spgarbet]

Spoke with Cole. His opinion was this all leads to complexity and is to be avoided. The default base method is just give an error for conditions that cannot proceed and a warning for something that isn't quite right but computation can proceed. Then the user can suppressWarnings or create a tryCatch block method of their own and filter them down.

[spgarbet]

After much work I have discovered that making this change has uncovered a series of bugs in the handling of cached meta information. This leads to tests sometimes failing and sometimes not when working interactively. (Aside, it truly isn't meta information but the specification of the data. REDCap calling this meta is abus de langage).

We have the following:

• meta
• arms
• events
• projectInformation
• mapping

What I need is a dependency graph so I can rework this. @nutterb If I change any one of those, what downstream is invalid if cached?

[nutterb]

This isn't very graphy, but it might get you what you're after. Helpful?

Also, it's based on filename, not method. should be pretty informative still

library(dplyr)
library(tidyr)

look_for <- c("rcon$arms()", 
              "rcon$events", 
              "rcon$metadata()", 
              "rcon$fieldnames()", 
              "rcon$mapping()", 
              "rcon$users()", 
              "rcon$user_roles()", 
              "rcon$user_role_assignment()", 
              "rcon$version()", 
              "rcon$projectInformation()", 
              "rcon$instruments()", 
              "rcon$fileRepository()", 
              "rcon$repeatInstrumentEvent()", 
              "rcon$dags()", 
              "rcon$dag_assignment()", 
              "rcon$externalCoding()")

files <- list.files("R", 
                    pattern = "\\.(r|R)$", 
                    full.names = TRUE)

has_match <- function(str, pattern, ...){
  any(grepl(pattern, str, ...))
}

Data <- expand.grid(look_for = look_for, 
                    filename = files, 
                    stringsAsFactors = FALSE) %>% 
  mutate(basename = basename(filename),
         patt_look_for = sub("[(]", "[(]", look_for),
         patt_look_for = sub("[)]", "[)]", look_for),
         patt_look_for = sub("[$]", "[$]", look_for),
         Code = lapply(filename, readLines), 
         has_match = mapply(has_match, 
                            Code, 
                            patt_look_for, 
                            SIMPLIFY = TRUE)) %>% 
  filter(has_match) %>% 
  arrange(look_for, basename) %>% 
  select(look_for, basename)


                       look_for                           basename
1                   rcon$arms()                       deleteArms.R
2                   rcon$arms()                    deleteRecords.R
3                   rcon$arms()                     exportEvents.R
4                   rcon$arms()            fieldCastingFunctions.R
5                   rcon$arms()                   importMappings.R
6                   rcon$dags()                       deleteDags.R
7                   rcon$dags()            fieldCastingFunctions.R
8                   rcon$dags()                       importDags.R
9                   rcon$dags()         importUserDagAssignments.R
10                  rcon$dags()                        switchDag.R
11                  rcon$events                      deleteFiles.R
12                  rcon$events                      exportFiles.R
13                  rcon$events                        exportPDF.R
14                  rcon$events                 exportProjectXml.R
15                  rcon$events                    exportRecords.R
16                  rcon$events               exportRecordsTyped.R
17                  rcon$events                 exportSurveyLink.R
18                  rcon$events         exportSurveyParticipants.R
19                  rcon$events           exportSurveyReturnCode.R
20                  rcon$events            fieldCastingFunctions.R
21                  rcon$events                      importFiles.R
22                  rcon$events                   importMappings.R
23        rcon$externalCoding()            fieldCastingFunctions.R
24            rcon$fieldnames()               exportRecordsTyped.R
25            rcon$fieldnames()               exportReportsTyped.R
26            rcon$fieldnames()            fieldCastingFunctions.R
27            rcon$fieldnames()                    importRecords.R
28            rcon$fieldnames()                 redcapConnection.R
29            rcon$fieldnames()                       splitForms.R
30        rcon$fileRepository()       createFileRepositoryFolder.R
31        rcon$fileRepository()         deleteFromFileRepository.R
32        rcon$fileRepository()               fileRepositoryPath.R
33        rcon$fileRepository()           importToFileRepository.R
34           rcon$instruments()                 assembleCodebook.R
35           rcon$instruments()                   exportMetaData.R
36           rcon$instruments()                        exportPDF.R
37           rcon$instruments()               exportRecordsTyped.R
38           rcon$instruments()                 exportSurveyLink.R
39           rcon$instruments()         exportSurveyParticipants.R
40           rcon$instruments()           exportSurveyReturnCode.R
41           rcon$instruments()            fieldCastingFunctions.R
42           rcon$instruments()                   importMappings.R
43           rcon$instruments()                      importUsers.R
44           rcon$instruments()               prepUserImportData.R
45           rcon$instruments()                       splitForms.R
46               rcon$mapping()                   importMappings.R
47              rcon$metadata()                  allocationTable.R
48              rcon$metadata()                 assembleCodebook.R
49              rcon$metadata()                      deleteFiles.R
50              rcon$metadata()             exportExternalCoding.R
51              rcon$metadata()                 exportFieldNames.R
52              rcon$metadata()                      exportFiles.R
53              rcon$metadata()                   exportMetaData.R
54              rcon$metadata()                    exportRecords.R
55              rcon$metadata()               exportRecordsTyped.R
56              rcon$metadata()                    exportReports.R
57              rcon$metadata()               exportReportsTyped.R
58              rcon$metadata()                      exportUsers.R
59              rcon$metadata()            fieldCastingFunctions.R
60              rcon$metadata()               getProjectIdFields.R
61              rcon$metadata()                      importFiles.R
62              rcon$metadata()                    importRecords.R
63              rcon$metadata()                   missingSummary.R
64              rcon$metadata()                 redcapConnection.R
65              rcon$metadata()                       splitForms.R
66    rcon$projectInformation()                 assembleCodebook.R
67    rcon$projectInformation()        constructLinkToRedcapForm.R
68    rcon$projectInformation()                      deleteFiles.R
69    rcon$projectInformation()                  dropRepeatingNA.R
70    rcon$projectInformation()                       exportArms.R
71    rcon$projectInformation()                     exportEvents.R
72    rcon$projectInformation()                      exportFiles.R
73    rcon$projectInformation()                   exportMappings.R
74    rcon$projectInformation() exportRepeatingInstrumentsEvents.R
75    rcon$projectInformation()               getProjectIdFields.R
76    rcon$projectInformation()                      importFiles.R
77    rcon$projectInformation()                  preserveProject.R
78 rcon$repeatInstrumentEvent()                 assembleCodebook.R
79  rcon$user_role_assignment()                      importUsers.R
80            rcon$user_roles()                  deleteUserRoles.R
81            rcon$user_roles()        importUserRoleAssignments.R
82            rcon$user_roles()                  importUserRoles.R
83                 rcon$users()                      deleteUsers.R
84                 rcon$users()         importUserDagAssignments.R
85                 rcon$users()        importUserRoleAssignments.R
86               rcon$version()        constructLinkToRedcapForm.R
87               rcon$version()                    exportRecords.R
88               rcon$version()                    exportReports.R
89               rcon$version()            fieldCastingFunctions.R
90               rcon$version()                    importRecords.R
91               rcon$version()                      importUsers.R
92               rcon$version()               prepUserImportData.R

[spgarbet]

That's a good view of it. I can distill that.

[spgarbet]

Closed with #348