Error when getting reference model with k-fold cross validation for cv_varsel

[adietzel]

I am revisiting a snippet of code in my analysis that used to work with previous versions of brms and projpred (Projpred Issue #328, and on a different machine after making switch from Windows to Mac.

My work flow with reprex below is:

1. Create custom stratiefied cross-validation folds with loo::kfold_split_stratified(),
2. Refit the model with CV using brms::kfold()
3. Create a custom reference model with the cvfits
4. Select relevant variables with cv_varsel()

The syntax used here for specifying a custom k-fold cv ref model is also shown in #1286

library(brms) # version 2.20.4
library(projpred) # version 2.7.0

myK <- 2

data <- data.frame(
  Pres = sample(c(0, 1), 100, replace = TRUE),
  X1 = rnorm(100, 0, 1),
  X2 = rnorm(100, 0.2, 1))

RefMod <- brm(
  formula = Pres ~ X1 + X2,
  family = bernoulli(),
  data = data)

ids <- loo::kfold_split_stratified(K = myK, x = data$Pres)
myfits_K <- brms::kfold(RefMod, folds = ids, K = myK, save_fits = TRUE)

RefMod_CV <- brms:::get_refmodel.brmsfit(
  object = RefMod,
  cvfits = structure(list("fits" = myfits_K$fits[, "fit"]),
                     "K" = myK, "folds" = ids))

mycvvs <- cv_varsel(
  RefMod_CV,
  method = "forward",
  cv_method = "kfold",
  K = myK,
  cvfits = RefMod_CV$cvfits,
  seed = 86132035)

When running cv_varsel() I get the following error message:

"in family$linkinv(pred_sub) :
Argument eta must be a nonempty numeric vector
In addition: Warning message:
In parse_args_cv_varsel(refmodel = refmodel, cv_method = cv_method, :
The content of cvfits's sub-list called fits should be moved one level up (and element fits removed). The old structure will continue to work for a while, but is deprecated."

I would appreciate any help with understanding the main error message and also with how to set the ref model correctly now.

Thanks for your help!

Andreas

[paul-buerkner]

I am not sure if this is rather a brms or projpred issue (https://github.com/stan-dev/projpred). Can you try if things are working with the latest brms and projpred versions from their respective github repos?

[adietzel]

I installed the github version of brms (2.20.6) but struggling to install the projpred github version due to issues with installing gfortran without admin rights on my computer. With brms updated but not projpred, I still run into the same issue. Should I post this issue on the github page of projpred?
Thanks

[paul-buerkner]

yes please also post it on the projpred page. as soon as we know where the problem lies we can close the other issue then. Andreas Dietzel ***@***.***> schrieb am Di., 12. Dez. 2023, 16:23:

…

I installed the github version of brms (2.20.6) but struggling to install the projpred github version due to issues with installing gfortran without admin rights on my computer. With brms updated but not projpred, I still run into the same issue. Should I post this issue on the github page of projpred? Thanks — Reply to this email directly, view it on GitHub <#1572 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADCW2AEGOWBREP7B6RQLDFDYJBSGJAVCNFSM6AAAAABARESXRCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJSGEZTKMZRGI> . You are receiving this because you commented.Message ID: ***@***.***>

[fweber144]

@paul-buerkner: You may close this issue here because it had already been fixed in projpred's GitHub version (commit stan-dev/projpred@b5bdc31).

[paul-buerkner]

Perfect. Thank you!