Error in sjSDM_cv

[YJ781]

Hi all,

I am running the model with GPU on the supercomputer and have the following error:

Error in cbind(nodes, (n_gpu - 1):0) : object 'nodes' not found
Calls: sjSDM_cv -> tune_func -> cbind

This is what I set for sjSDM_cv
(env = as.matrix(train_X), Y = as.matrix(train_Y),
learning_rate = 0.01, iter = 100L, CV = 10,
tune_steps = 40,
lambda_cov = seq(0, 0.1, 0.001),
lambda_coef = seq(0, 0.1, 0.001),
alpha_cov = seq(0, 1, 0.05),
alpha_coef = seq(0, 1, 0.05),
device = "gpu",
n_gpu = 1,
sampling = 100L,
biotic = bioticStruct(df=dim(train_Y)[2]),
blocks = 6L,
step_size = 4L,
family=gaussian("identity")
)

Do you know how I can figure this out?
Thanks!

[MaximilianPi]

Hi @YJ781,

If you set n_gpu = 1L , sjSDM_cv is expecting a CPU parallelization which you didn't specify, so either you turn on CPU parallelization by using n_cores = X or you set n_gpu = NULL (n_gpu = NULL and n_gpu = 1 is actually the same, in both cases only one GPU is used).

Thanks for reporting, I will implement a check for that.

[YJ781]

Hi @MaximilianPi ,
Thank you for your explanations. I've tried two settings but got errors as well. Here are the details of the error messages.

n_gpu = NULL
Error in py_call_impl(callable, call_args$unnamed, call_args$named) :
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
Run reticulate::py_last_error() for details.
Calls: sjSDM_cv ... sjSDM -> system.time -> -> py_call_impl
Timing stopped at: 0.494 0.223 0.719
Execution halted

n_gpu=2
Error in cbind(nodes, (n_gpu - 1):0) : object 'nodes' not found
Calls: sjSDM_cv -> tune_func -> cbind
Execution halted

[MaximilianPi]

Hi @YJ781,

This is a bug, I forgot to move a tensor to the right device.

I pushed a fix to the development branch (master branch of the github repository), you can install the development version by running the following:
devtools::install_github("https://github.com/TheoreticalEcology/s-jSDM", subdir = "sjSDM", ref = "master")

[YJ781]

Hi @MaximilianPi ,

Thank you so much. It worked.

But for n_gpu=2, I'm having the same error,

Error in cbind(nodes, (n_gpu - 1):0) : object 'nodes' not found
Calls: sjSDM_cv -> tune_func -> cbind
Execution halted

How can I deal with it?

[MaximilianPi]

Hi @YJ781,

Do you still have this error?

[YJ781]

Yes. Same error.

…

________________________________ From: MaximilianPi ***@***.***> Sent: Tuesday, November 14, 2023 5:21:27 AM To: TheoreticalEcology/s-jSDM ***@***.***> Cc: YJ781 ***@***.***>; Mention ***@***.***> Subject: Re: [TheoreticalEcology/s-jSDM] Error in sjSDM_cv (Issue #131) Hi @YJ781<https://github.com/YJ781>, Do you still have this error? — Reply to this email directly, view it on GitHub<#131 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AQ4SGZIXH32WTN6K5SQS4B3YENH3PAVCNFSM6AAAAAA5GX3EKWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJQGAYTSMJWGM>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[YJ781]

Hi @MaximilianPi ,

I feel sorry I forgot to @ you in the last comment. I'm still having the same error. Thanks!

[MaximilianPi]

Hi @YJ781,
I will look into this!