Inconsistent treatment of unicode character in RStudio vs quarto

[richardjtelford]

This is a problem that affects some of the students with windows computers in my class.

The students are importing an excel file that contains a unicode character \u2103 (℃) in the header row. They are then using janitor::clean_names().

For most of the students janitor::clean_names() converts the column name to "temperature_c" in both Rstudio and when rendering with quarto.
For about 20% of the students, janitor::clean_names() converts the "℃" to "temperature_u_00b0_c" (the unicode for "°") in Rstudio but to "temperature_c" when rendered with quarto. This then causes problems with the rest of their code when they render the document

In both rstudio and quarto the "℃" is being imported correctly as utf-8 and has the same output with charToRaw() - e2 84 83, so it is not an import problem. Somehow janitor is treating the unicode differently depending on how R is being run.

All the affected students are using R4.2.1 with the current version of RStudio on windows. Students might have Norwegian locales - I haven't been able to check that.

Minimal example (but it might work correctly for you)

tibble::tibble("Temperature (℃)" = 1) |> janitor::clean_names() |> names()
#temperature_u_00b0_c 

[billdenney]

Can you please share the sessionInfo() from one system with the extra text and one without?

Having different language settings can change the way that characters are rendered and converted. We have worked hard to make it consistent, but it seems another case has snuck through.

Sometimes, other libraries will do some degree of transliteration before it gets to clean_names(), too. What library and function are you using to load the excel file?

[richardjtelford]

We are importing the data with readxl::read_excel().

Here is the sessionInfo from one student where unicode went wrong. I think the other students with the problem had the same locale.

R version 4.2.1 (2022-06-23 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19043)


Matrix products: default

 

locale:

[1] C

system code page: 65001

 

attached base packages:

[1] stats     graphics  grDevices utils     datasets  methods   base    

 

other attached packages:

[1] janitor_2.1.0   readxl_1.4.1    forcats_0.5.2   stringr_1.4.1   dplyr_1.0.10    purrr_0.3.4     readr_2.1.2     tidyr_1.2.1   
 [9] tibble_3.1.8    ggplot2_3.3.6   tidyverse_1.3.2 here_1.0.1      tidylog_1.0.2 

 

loaded via a namespace (and not attached):

[1] Rcpp_1.0.9          lubridate_1.8.0     lattice_0.20-45     class_7.3-20        clisymbols_1.2.0    digest_0.6.29       assertthat_0.2.1  
 [8] rprojroot_2.0.3     utf8_1.2.2          R6_2.5.1            cellranger_1.1.0    backports_1.4.1     reprex_2.0.2        evaluate_0.16     
[15] e1071_1.7-11        httr_1.4.4          pillar_1.8.1        rlang_1.0.5         googlesheets4_1.0.1 rstudioapi_0.14     Matrix_1.5-1      
[22] rmarkdown_2.16      labeling_0.4.2      splines_4.2.1       googledrive_2.0.0   munsell_0.5.0       proxy_0.4-27        broom_1.0.1       
[29] compiler_4.2.1      modelr_0.1.9        xfun_0.34           pkgconfig_2.0.3     mgcv_1.8-40         htmltools_0.5.3     tidyselect_1.1.2  
[36] fansi_1.0.3         crayon_1.5.1        tzdb_0.3.0          dbplyr_2.2.1        withr_2.5.0         grid_4.2.1          nlme_3.1-157      
[43] jsonlite_1.8.0      gtable_0.3.1        lifecycle_1.0.2     DBI_1.1.3           magrittr_2.0.3      units_0.8-0         scales_1.2.1      
[50] KernSmooth_2.23-20  cli_3.4.0           stringi_1.7.8       farver_2.1.1        fs_1.5.2            snakecase_0.11.0    xml2_1.3.3        
[57] ellipsis_0.3.2      generics_0.1.3      vctrs_0.4.1         tools_4.2.1         glue_1.6.2          hms_1.1.2           yaml_2.3.5        
[64] fastmap_1.1.0       colorspace_2.0-3    gargle_1.2.0        classInt_0.4-8      rvest_1.0.3         knitr_1.40          haven_2.5.1   

Here is the session info from a student who had it working as expected

> sessionInfo()
R version 4.2.1 (2022-06-23 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 22000)

Matrix products: default

locale:
[1] LC_COLLATE=English_United Kingdom.utf8  LC_CTYPE=English_United Kingdom.utf8  
[3] LC_MONETARY=English_United Kingdom.utf8 LC_NUMERIC=C                          
[5] LC_TIME=English_United Kingdom.utf8    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base    

other attached packages:
 [1] ggfortify_0.4.14 gt_0.7.0         readxl_1.4.1     broom_1.0.1      forcats_0.5.2    stringr_1.4.1  
 [7] dplyr_1.0.9      purrr_0.3.4      readr_2.1.2      tidyr_1.2.0      tibble_3.1.8     ggplot2_3.3.6  
[13] tidyverse_1.3.2

loaded via a namespace (and not attached):
 [1] lubridate_1.8.0     assertthat_0.2.1    digest_0.6.29       utf8_1.2.2          R6_2.5.1          
 [6] cellranger_1.1.0    backports_1.4.1     reprex_2.0.2        evaluate_0.16       httr_1.4.4        
[11] highr_0.9           pillar_1.8.1        rlang_1.0.4         googlesheets4_1.0.1 rstudioapi_0.14    
[16] car_3.1-0           labeling_0.4.2      googledrive_2.0.0   bit_4.0.4           munsell_0.5.0      
[21] compiler_4.2.1      modelr_0.1.9        janitor_2.1.0       xfun_0.32           pkgconfig_2.0.3    
[26] htmltools_0.5.3     tidyselect_1.1.2    gridExtra_2.3       fansi_1.0.3         crayon_1.5.1      
[31] tzdb_0.3.0          dbplyr_2.2.1        withr_2.5.0         grid_4.2.1          jsonlite_1.8.0    
[36] gtable_0.3.0        lifecycle_1.0.1     DBI_1.1.3           magrittr_2.0.3      scales_1.2.1      
[41] cli_3.4.1           stringi_1.7.8       vroom_1.5.7         carData_3.0-5       farver_2.1.1      
[46] fs_1.5.2            snakecase_0.11.0    xml2_1.3.3          ellipsis_0.3.2      generics_0.1.3    
[51] vctrs_0.4.1         RColorBrewer_1.1-3  tools_4.2.1         bit64_4.0.5         glue_1.6.2        
[56] hms_1.1.2           fastmap_1.1.0       abind_1.4-5         parallel_4.2.1      colorspace_2.0-3  
[61] gargle_1.2.0        rvest_1.0.3         knitr_1.40          haven_2.5.1         sass_0.4.2  

[billdenney]

I was looking at this again today, and it's an even thornier problem than I first thought. When I tried it on my system (Windows 11 with US English locale), I get a simple lowercase c (see below). But, I think that a workaround can be using the replace argument:

tibble::tibble("Temperature (℃)" = 1) |> janitor::clean_names(replace = c("\u2103" = "deg c")) |> names()

degC <- rawToChar(as.raw(c(0xe2, 0x84, 0x83)))
degC
#> [1] "℃"
janitor::make_clean_names(degC)
#> [1] "c"

^{Created on 2022-12-01 with reprex v2.0.2}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.2.1 (2022-06-23 ucrt)
#>  os       Windows 10 x64 (build 22621)
#>  system   x86_64, mingw32
#>  ui       RTerm
#>  language (EN)
#>  collate  English_United States.utf8
#>  ctype    English_United States.utf8
#>  tz       America/New_York
#>  date     2022-12-01
#>  pandoc   2.19.2 @ C:/Program Files/RStudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version date (UTC) lib source
#>  assertthat    0.2.1   2019-03-21 [1] CRAN (R 4.2.1)
#>  cli           3.4.1   2022-09-23 [1] CRAN (R 4.2.1)
#>  DBI           1.1.3   2022-06-18 [1] CRAN (R 4.2.1)
#>  digest        0.6.29  2021-12-01 [1] CRAN (R 4.2.1)
#>  dplyr         1.0.10  2022-09-01 [1] CRAN (R 4.2.1)
#>  evaluate      0.18    2022-11-07 [1] CRAN (R 4.2.2)
#>  fansi         1.0.3   2022-03-24 [1] CRAN (R 4.2.1)
#>  fastmap       1.1.0   2021-01-25 [1] CRAN (R 4.2.1)
#>  fs            1.5.2   2021-12-08 [1] CRAN (R 4.2.1)
#>  generics      0.1.3   2022-07-05 [1] CRAN (R 4.2.1)
#>  glue          1.6.2   2022-02-24 [1] CRAN (R 4.2.1)
#>  highr         0.9     2021-04-16 [1] CRAN (R 4.2.1)
#>  htmltools     0.5.3   2022-07-18 [1] CRAN (R 4.2.1)
#>  janitor       2.1.0   2021-01-05 [1] CRAN (R 4.2.1)
#>  knitr         1.40    2022-08-24 [1] CRAN (R 4.2.1)
#>  lifecycle     1.0.3   2022-10-07 [1] CRAN (R 4.2.1)
#>  lubridate     1.8.0   2021-10-07 [1] CRAN (R 4.2.1)
#>  magrittr      2.0.3   2022-03-30 [1] CRAN (R 4.2.1)
#>  pillar        1.8.1   2022-08-19 [1] CRAN (R 4.2.1)
#>  pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.2.1)
#>  purrr         0.3.5   2022-10-06 [1] CRAN (R 4.2.1)
#>  R.cache       0.16.0  2022-07-21 [1] CRAN (R 4.2.1)
#>  R.methodsS3   1.8.2   2022-06-13 [1] CRAN (R 4.2.0)
#>  R.oo          1.25.0  2022-06-12 [1] CRAN (R 4.2.0)
#>  R.utils       2.12.0  2022-06-28 [1] CRAN (R 4.2.1)
#>  R6            2.5.1   2021-08-19 [1] CRAN (R 4.2.1)
#>  reprex        2.0.2   2022-08-17 [1] CRAN (R 4.2.1)
#>  rlang         1.0.6   2022-09-24 [1] CRAN (R 4.2.1)
#>  rmarkdown     2.17    2022-10-07 [1] CRAN (R 4.2.1)
#>  rstudioapi    0.14    2022-08-22 [1] CRAN (R 4.2.1)
#>  sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.2.1)
#>  snakecase     0.11.0  2019-05-25 [1] CRAN (R 4.2.1)
#>  stringi       1.7.8   2022-07-11 [1] CRAN (R 4.2.1)
#>  stringr       1.4.1   2022-08-20 [1] CRAN (R 4.2.1)
#>  styler        1.7.0   2022-03-13 [1] CRAN (R 4.2.1)
#>  tibble        3.1.8   2022-07-22 [1] CRAN (R 4.2.1)
#>  tidyselect    1.2.0   2022-10-10 [1] CRAN (R 4.2.1)
#>  utf8          1.2.2   2021-07-24 [1] CRAN (R 4.2.1)
#>  vctrs         0.5.0   2022-10-22 [1] CRAN (R 4.2.2)
#>  withr         2.5.0   2022-03-03 [1] CRAN (R 4.2.1)
#>  xfun          0.34    2022-10-18 [1] CRAN (R 4.2.2)
#>  yaml          2.3.6   2022-10-18 [1] CRAN (R 4.2.1)
#> 
#>  [1] C:/Users/wdenn/AppData/Local/R/win-library/4.2
#>  [2] C:/Program Files/R/R-4.2.1/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

[billdenney]

I'm not sure what the best solution within janitor is for this. A simple solution would be to augment the default replace argument with this character. But, I think that would end up taking us down a path of making our own Unicode transliteration list (which is not something that this package has the capacity to keep up with).

Ideas are welcome.