2018 non-functionality

[erex]

Error arises when doing conversion of (allegedly) simple project (songbird, filtering for song sparrow, w=68)

Tracked error to function unit_tab()

Error in `$<-.data.frame`(`*tmp*`, Conversion, value = numeric(0)) : 
  replacement has 0 rows, data has 74
In addition: Warning message:
Unknown or uninitialised column: 'Conversion'. 

The beast created by read.delim() trick is messy. Solution appears to be this:

Browse[2]> str(unit_tab)
Classes ‘tbl_df’, ‘tbl’ and 'data.frame':	74 obs. of  2 variables:
 $                  Unit     : chr  "                     Degree  " "                        Gon  " "                       Grad  " "                Microradian  " ...
 $      Conversion           : chr  " 1.745329251994329547437e-02" " 1.570796326794896696777e-02" " 1.570796326794896696777e-02" " 9.999999999999999547481e-07" ...
 - attr(*, "spec")=List of 2
  ..$ cols   :List of 2
  .. ..$                  Unit     : list()
  .. .. ..- attr(*, "class")= chr  "collector_character" "collector"
  .. ..$      Conversion           : list()
  .. .. ..- attr(*, "class")= chr  "collector_character" "collector"
  ..$ default: list()
  .. ..- attr(*, "class")= chr  "collector_guess" "collector"
  ..- attr(*, "class")= chr "col_spec"
Browse[2]> names(unit_tab)
[1] "                 Unit     " "     Conversion"           
Browse[2]> names(unit_tab) <- c("Unit", "Conversion")
Browse[2]> n
debug: return(unit_tab)

I'll made that change and commit

[erex]

yes, there are no strata areas in songbird project, so abundance cannot be computed, but many other things break when trying to duplicate analysis in which size-biased regression by stratum is requested:

> fred <- test_stats(tmp[[1]], statuses=2)
Error in create.varstructure(model, region.table, sample.table, obs.table) : 
  Invalid or missing Area values for regions
In addition: Warning messages:
1: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
2: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
3: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
4: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
5: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
6: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
7: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
8: In create.varstructure(model, region.table, sample.table, obs.table) :
  Some samples not included in the analysis

no object is created

quite an exotic P_a is computed from the fitted detection function:

                    Estimate       SE        CV
Average p            4.53580  2.66565 0.5876912

[erex]

Still working with songbird project, but trying second analysis with cluster size as detection function covariate:
somewhere (don't know what function does this), the column "Cluster size" does not have the space replaced by a full stop. No problems arise from convert_project()

But run_analysis() fails with:

> two <- run_analysis(tmp[[2]])
Error in parse(text = analysis$call) : <text>:1:52: unexpected symbol
1: mrds::ddf(dsmodel=~mcds(key="hn", formula=~Cluster size

[dill]

Contrary to my confidence this afternoon, I can't reproduce step one here where the Conversion column appears to be read in as character rather than numeric. Can you give me the output of sessionInfo()?

[erex]

still the same result:

> library(readdst)
> tmp <- convert_project("C:\\Users\\erexstad\\Desktop\\readdst-test-projects\\songbird")
Loading required package: RODBC
> one <- run_analysis(tmp[[1]])
> fred <- test_stats(tmp[[1]], statuses=2)
Error in create.varstructure(model, region.table, sample.table, obs.table) : 
  Invalid or missing Area values for regions
In addition: Warning messages:
1: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
2: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
3: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
4: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
5: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
6: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
7: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
8: In create.varstructure(model, region.table, sample.table, obs.table) :
  Some samples not included in the analysis
> sessionInfo()
R version 3.4.3 (2017-11-30)
Platform: i386-w64-mingw32/i386 (32-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United Kingdom.1252  LC_CTYPE=English_United Kingdom.1252   
[3] LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C                           
[5] LC_TIME=English_United Kingdom.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] RODBC_1.3-15  readdst_0.0.4

loaded via a namespace (and not attached):
 [1] RColorBrewer_1.1-2  checkmate_1.8.5     cluster_2.0.6       rstudioapi_0.7     
 [5] magrittr_1.5        acepack_1.4.1       gtable_0.2.0        minqa_1.2.4        
 [9] data.table_1.10.4-3 base64enc_0.1-3     pillar_1.2.1        htmltools_0.3.6    
[13] stringr_1.3.0       truncnorm_1.0-8     splines_3.4.3       Formula_1.2-2      
[17] lattice_0.20-35     survival_2.41-3     Rvmmin_2017-7.18    setRNG_2013.9-1    
[21] htmlwidgets_1.0     testthat_2.0.0      plyr_1.8.4          knitr_1.20         
[25] gridExtra_2.3       Matrix_1.2-12       R6_2.2.2            optimx_2013.8.7    
[29] numDeriv_2016.8-1   digest_0.6.15       colorspace_1.3-2    Rsolnp_1.16        
[33] stringi_1.1.6       yaml_2.1.18         lazyeval_0.2.1      Hmisc_4.1-1        
[37] dfoptim_2017.12-1   tibble_1.4.2        mgcv_1.8-23         compiler_3.4.3     
[41] Rcgmin_2013-2.21    rpart_4.1-11        backports_1.1.2     htmlTable_1.11.2   
[45] munsell_0.4.3       Rcpp_0.12.15        optextras_2016-8.8  BB_2014.10-1       
[49] svUnit_0.7-12       parallel_3.4.3      ggplot2_2.2.1       ucminf_1.1-4       
[53] mrds_2.1.18         latticeExtra_0.6-28 tools_3.4.3         foreign_0.8-69     
[57] nnet_7.3-12         scales_0.5.0        quadprog_1.5-5      rlang_0.2.0        
[61] nlme_3.1-131.1      grid_3.4.3         

[dill]

Odd. I wonder if this is a Windows issue? I'll try to source some way of testing that. In the meantime can you upload the project here so I can double check?

…

On 10/03/2018 10:52, erex wrote: still the same result: | > library(readdst) > tmp <- convert_project("C:\\Users\\erexstad\\Desktop\\readdst-test-projects\\songbird") Loading required package: RODBC > one <- run_analysis(tmp[[1]]) > fred <- test_stats(tmp[[1]], statuses=2) Error in create.varstructure(model, region.table, sample.table, obs.table) : Invalid or missing Area values for regions In addition: Warning messages: 1: In predict.lm(size_lm, pred_data) : prediction from a rank-deficient fit may be misleading 2: In predict.lm(size_lm, pred_data) : prediction from a rank-deficient fit may be misleading 3: In predict.lm(size_lm, pred_data) : prediction from a rank-deficient fit may be misleading 4: In predict.lm(size_lm, pred_data) : prediction from a rank-deficient fit may be misleading 5: In predict.lm(size_lm, pred_data) : prediction from a rank-deficient fit may be misleading 6: In predict.lm(size_lm, pred_data) : prediction from a rank-deficient fit may be misleading 7: In predict.lm(size_lm, pred_data) : prediction from a rank-deficient fit may be misleading 8: In create.varstructure(model, region.table, sample.table, obs.table) : Some samples not included in the analysis > sessionInfo() R version 3.4.3 (2017-11-30) Platform: i386-w64-mingw32/i386 (32-bit) Running under: Windows >= 8 x64 (build 9200) Matrix products: default locale: [1] LC_COLLATE=English_United Kingdom.1252 LC_CTYPE=English_United Kingdom.1252 [3] LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C [5] LC_TIME=English_United Kingdom.1252 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] RODBC_1.3-15 readdst_0.0.4 loaded via a namespace (and not attached): [1] RColorBrewer_1.1-2 checkmate_1.8.5 cluster_2.0.6 rstudioapi_0.7 [5] magrittr_1.5 acepack_1.4.1 gtable_0.2.0 minqa_1.2.4 [9] data.table_1.10.4-3 base64enc_0.1-3 pillar_1.2.1 htmltools_0.3.6 [13] stringr_1.3.0 truncnorm_1.0-8 splines_3.4.3 Formula_1.2-2 [17] lattice_0.20-35 survival_2.41-3 Rvmmin_2017-7.18 setRNG_2013.9-1 [21] htmlwidgets_1.0 testthat_2.0.0 plyr_1.8.4 knitr_1.20 [25] gridExtra_2.3 Matrix_1.2-12 R6_2.2.2 optimx_2013.8.7 [29] numDeriv_2016.8-1 digest_0.6.15 colorspace_1.3-2 Rsolnp_1.16 [33] stringi_1.1.6 yaml_2.1.18 lazyeval_0.2.1 Hmisc_4.1-1 [37] dfoptim_2017.12-1 tibble_1.4.2 mgcv_1.8-23 compiler_3.4.3 [41] Rcgmin_2013-2.21 rpart_4.1-11 backports_1.1.2 htmlTable_1.11.2 [45] munsell_0.4.3 Rcpp_0.12.15 optextras_2016-8.8 BB_2014.10-1 [49] svUnit_0.7-12 parallel_3.4.3 ggplot2_2.2.1 ucminf_1.1-4 [53] mrds_2.1.18 latticeExtra_0.6-28 tools_3.4.3 foreign_0.8-69 [57] nnet_7.3-12 scales_0.5.0 quadprog_1.5-5 rlang_0.2.0 [61] nlme_3.1-131.1 grid_3.4.3 | — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#34 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAHoRJCrCKM2cSvI_IZTxBJQVCTruueks5tc7BlgaJpZM4Si1kg>.

-- PS. I am currently engaged in industrial action (university pensions are under threat in the UK, see https://www.ucu.org.uk/why-we-are-taking-action-over-USS) so I may be very slow to reply during the coming weeks.

[erex]

as requested
Songbird.zip

[dill]

hmm, okay, can you ensure you have the latest readdst from github?

I am now getting the warnings that you get, now that I am using this data set:

Warning messages:
1: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
2: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
3: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
4: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
5: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
6: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
7: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
8: In dht(model, obs.table = analysis$env$obs.table, sample.table = analysis$env$sample.table,  :
  Point transect encounter rate variance can only use estimator P3, switching to this estimator.
9: In ticks[as.numeric(res_text) >= tolerance] <- "" :
  NAs introduced by coercion
10: In data.frame(Statistic = stats$Name, Distance_value = as.numeric(stats$Value),  :
  NAs introduced by coercion

but not the error. When we looked at this in your office I don't recall seeing the as.numeric at line 90 of units_table() but I'm guessing that's my memory. I'll look further into the dht() call just now...

[dill]

Okay, one possible issue here:

print(analysis$env$region.table)
  Region.Label Area
1       1980_1    0
2       1980_2    0
3       1980_3    0
4       1981_0    0
5       1981_1    0
6       1981_2    0
7       1981_3    0

wonder why this doesn't cause an error for me but does for you... mrds version perhaps? According to the above you have mrds_2.1.18, where as I am running the development version from github (displaying as 2.2.0).

In this case (where areas are 0) I guess we should not return comparisons of Nhat? What does Distance do in this situation?

FYI, the output of running the above for me:

        Statistic Distance_value    mrds_value   Difference Pass
1               n   3.110000e+02  3.110000e+02 0.000000e+00    ✓
2      parameters   2.000000e+00  2.000000e+00 0.000000e+00    ✓
3             AIC   2.462557e+03  2.462518e+03 1.564564e-05    ✓
4         Chi^2 p   7.224567e-01  7.997105e-01 1.069321e-01
5             P_a   1.409893e-01  1.409634e-01 1.833682e-04    ✓
6         CV(P_a)   9.722771e-02  9.722315e-02 4.685437e-05    ✓
7  log-likelihood  -1.229278e+03 -1.229259e+03 0.000000e+00    ✓
8           K-S p   9.291519e-01            NA           NA    ✓
9          C-vM p   1.000000e+00  9.000000e-01 1.000000e-01
10        density   2.115865e+00  5.972563e+04 2.822652e+04
11    CV(density)   1.125527e-01  1.005158e-01 1.069441e-01

[erex]

Guess I'll grab a 2.2.0 version of mrds and see what happens. Appears to have gotten the version bump in October of last year when you were dealing with the K-S test revamp; with estimated values of the distribution parameters

Without stratum areas provided, Distance 7 simply omits any attempt to produce abundance estimates and only reports density estimates:

                         Estimate      %CV     df     95% Confidence Interval
                        ------------------------------------------------------
 Stratum: 1. 1980_1                                         
 Half-normal/Cosine     
                 DS      1.9530       17.05   239.93  1.3993       2.7259    
                 D       1.9530       17.05   239.93  1.3993       2.7259    
 Stratum: 2. 1980_2                                         
 Half-normal/Cosine     
                 DS      1.9921       17.89   233.29  1.4042       2.8260    
                 D       2.1464       18.06   241.70  1.5082       3.0545    
 Stratum: 3. 1980_3                                         
 Half-normal/Cosine     
                 DS      2.0245       17.64   234.17  1.4341       2.8580    
                 D       2.2124       17.74   239.75  1.5640       3.1296    
 Stratum: 4. 1981_0                                         
 Half-normal/Cosine     
                 DS      2.0995       17.46   195.32  1.4917       2.9550    
                 D       2.2740       17.73   206.50  1.6077       3.2166    
 Stratum: 5. 1981_1                                         
 Half-normal/Cosine     
                 DS      2.3436       16.82   206.64  1.6861       3.2575    
                 D       2.7021       16.94   212.66  1.9394       3.7648    
 Stratum: 6. 1981_2                                         
 Half-normal/Cosine     
                 DS      1.9530       17.22   199.34  1.3941       2.7360    
                 D       2.0436       17.32   203.87  1.4560       2.8682    
 Stratum: 7. 1981_3                                         
 Half-normal/Cosine     
                 DS      1.4796       21.01   155.03 0.98134       2.2307    
                 D       1.4796       21.01   155.03 0.98134       2.2307    

[erex]

results using mrds 2.2.0

btw: the 'as.numeric()' statement did exist in earlier versions of readdst

> library(readdst)
> tmp <- convert_project("C:\\Users\\erexstad\\Desktop\\readdst-test-projects\\songbird")
Loading required package: RODBC
> one <- run_analysis(tmp[[1]])
> fred <- test_stats(tmp[[1]], statuses=2)
Error in create.varstructure(model, region.table, sample.table, obs.table) : 
  Invalid or missing Area values for regions
In addition: Warning messages:
1: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
2: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
3: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
4: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
5: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
6: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
7: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
8: In create.varstructure(model, region.table, sample.table, obs.table) :
  Some samples not included in the analysis
> sessionInfo()
R version 3.4.3 (2017-11-30)
Platform: i386-w64-mingw32/i386 (32-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United Kingdom.1252  LC_CTYPE=English_United Kingdom.1252    LC_MONETARY=English_United Kingdom.1252
[4] LC_NUMERIC=C                            LC_TIME=English_United Kingdom.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] RODBC_1.3-15  readdst_0.0.4

loaded via a namespace (and not attached):
 [1] RColorBrewer_1.1-2  checkmate_1.8.5     cluster_2.0.6       rstudioapi_0.7      magrittr_1.5        acepack_1.4.1      
 [7] gtable_0.2.0        memoise_1.1.0       minqa_1.2.4         data.table_1.10.4-3 base64enc_0.1-3     pillar_1.2.1       
[13] htmltools_0.3.6     stringr_1.3.0       curl_3.1            truncnorm_1.0-8     splines_3.4.3       Formula_1.2-2      
[19] lattice_0.20-35     survival_2.41-3     Rvmmin_2017-7.18    setRNG_2013.9-1     htmlwidgets_1.0     testthat_2.0.0     
[25] plyr_1.8.4          knitr_1.20          git2r_0.21.0        gridExtra_2.3       Matrix_1.2-12       R6_2.2.2           
[31] optimx_2013.8.7     numDeriv_2016.8-1   digest_0.6.15       colorspace_1.3-2    devtools_1.13.5     Rsolnp_1.16        
[37] stringi_1.1.6       dfoptim_2017.12-1   yaml_2.1.18         lazyeval_0.2.1      Hmisc_4.1-1         tibble_1.4.2       
[43] mgcv_1.8-23         httr_1.3.1          compiler_3.4.3      Rcgmin_2013-2.21    withr_2.1.1         rpart_4.1-11       
[49] backports_1.1.2     htmlTable_1.11.2    munsell_0.4.3       Rcpp_0.12.15        optextras_2016-8.8  BB_2014.10-1       
[55] svUnit_0.7-12       parallel_3.4.3      ggplot2_2.2.1       ucminf_1.1-4        mrds_2.2.0          latticeExtra_0.6-28
[61] tools_3.4.3         foreign_0.8-69      nnet_7.3-12         scales_0.5.0        quadprog_1.5-5      rlang_0.2.0        
[67] nlme_3.1-131.1      grid_3.4.3         

[dill]

Okay, well that's annoying that didn't fix things. On the other hand, I do think your version of mrds is doing the right thing: attempting to calculate abundance from zero areas should throw an error BUT readdst should catch that... Will write some code to not ask dht impossible questions in readdst::test_stats.

[dill]

26b0688 gets around this error, but do need to implement a density estimator (for groups and individuals and their respective s.e.s) when dht isn't called.

[erex]

Indeed the most recent commit does sneak around the error trying to produce the abundance estimate in test_stats() However, stepping back to simply looking at the results produced by run_analysis() is unusual:

> summary(out1)

Summary for ds object 
Number of observations :  311 
Distance range         :  0  -  68 
AIC                    :  2466.627 

Detection function:
 Half-normal key function with cosine adjustment terms of order 2,3,4,5 

Detection function parameters 
Scale coefficient(s):  
            estimate         se
(Intercept) 3.362437 0.06409272

Adjustment term coefficient(s):  
               estimate         se
cos, order 2 -0.3275389 0.10075289
cos, order 3 -0.1880255 0.12151259
cos, order 4 -0.1667302 0.10897364
cos, order 5 -0.2034401 0.08989036

                    Estimate       SE        CV
Average p            4.53580  2.66565 0.5876912
N in covered region 68.56563 39.62665 0.5779376

The D7-GUI fits a single adjustment term, whereas mrds fits 4 adjustment terms, (having a reasonably similar log-likelihood) with the end result being absurd estimate of detection probability:

> out2 <- test_stats(tmp[[1]], statuses=2)
Warning messages:
1: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
2: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
3: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
4: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
5: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
6: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
7: In predict.lm(size_lm, pred_data) :
  prediction from a rank-deficient fit may be misleading
8: In ticks[as.numeric(res_text) >= tolerance] <- "" :
  NAs introduced by coercion
9: In data.frame(Statistic = stats$Name, Distance_value = as.numeric(stats$Value),  :
  NAs introduced by coercion
> print(out2)
       Statistic Distance_value    mrds_value     Difference     Pass
1              n            311           311              0 <U+2713>
2     parameters              2             5            1.5         
3            AIC 2462.556884766 2466.62744576    0.001652935 <U+2713>
4        Chi^2 p      0.7224567      0.424512      0.4124049         
5            P_a      0.1409893     4.5358003       31.17124         
6        CV(P_a)     0.09722771    0.58769121       5.044483         
7 log-likelihood      -1229.278     -1228.314              0 <U+2713>
8          K-S p      0.9291519            NA             NA <U+2713>
9         C-vM p              1             1              0 <U+2713>

[erex]

20Mar efforts

House wren sample project: point transects, multiple visits, with observer as potential covariate

tmp <- convert_project("C:\\Users\\erexstad\\Desktop\\readdst-test-projects\\house wren")
one <- run_analysis(tmp[[1]])
two <- run_analysis(tmp[[2]])
fred <- test_stats(tmp[[1]], statuses=2)
> summary(one)

Summary for ds object 
Number of observations :  212 
Distance range         :  0  -  50 
AIC                    :  1641.048 

Detection function:
 Half-normal key function with cosine adjustment term of order 2 

Detection function parameters 
Scale coefficient(s):  
            estimate        se
(Intercept) 3.729443 0.1957697

Adjustment term coefficient(s):  
               estimate        se
cos, order 2 -0.3535751 0.1037593

                      Estimate       SE        CV
Average p             1.254026  0.23332 0.1860567
N in covered region 169.055460 30.90474 0.1828083
> two <- run_analysis(tmp[[2]])
Error in `contrasts<-`(`*tmp*`, value = contr.funs[1 + isOF[nn]]) : 
  contrasts can be applied only to factors with 2 or more levels
> fred <- test_stats(tmp[[1]], statuses=2)
Error in data.frame(Region.Label = names(Effort.by.region), CoveredArea = CoveredArea,  : 
  arguments imply differing number of rows: 10, 0

• project seems to be read properly
• computation of detection function parameter estimates is wrong (p-hat>1)
• analysis 2 (with observer factor covariate) fails because of 'contrasts()'
• test.stats() also fails for the simple half-normal model