Help debugging with SLURM

[Aariq]

Ever since my institution upgraded their cluster OS to RHEL9, I haven't been able to get crew.cluster to work and the error message I get in the worker logs points to a mirai function. They all end something like this:

Error in dial(sock, url = url, autostart = asyncdial || NA, tls = tls,  : 
  31 | Connection shutdown
Calls: <Anonymous> ... do.call -> <Anonymous> -> dial_and_sync_socket -> dial
Execution halted

I'm trying to come up with a reprex using just mirai to try to understand why this is happening and enable the HPC people to help fix this.

If I follow the mirai documentation and run the following code, two SLURM jobs are submitted, but as soon as they are started they immediately stop.

daemons(
  n = 2,
  url = host_url(tls=TRUE),
  remote = remote_config(
    command = "sbatch",
    args = c("--mem 512", "-n 1", "--wrap", "."),
    rscript = file.path(R.home("bin"), "Rscript"),
    quote = TRUE
  )
)

They create log files that look like so:

Error in dial(sock, url = url, autostart = asyncdial || NA, tls = tls,  : 
  14 | Destination unreachable
Calls: <Anonymous> -> dial_and_sync_socket -> dial
Execution halted

Which is not quite the same error message ("destination unreachable" rather than "connection shutdown"). I'd appreciate any guidance on how to re-create what crew.cluster is doing an any advice to make these logs more verbose so that the HPC professionals can diagnose the issue.

[shikokuchuo]

Thanks for reporting these. That error 14 destination unreachable is pretty rare. Can you try these first in case they work for you:

1. To your daemons() call add an argument asyncdial = TRUE. This will be passed to daemon() on the cluster nodes.
2. Instead of using host_url(tls=TRUE), substitute the hostname for the actual IP address of the login node you're using.

[shikokuchuo]

With asyncdial = TRUE the SLURM jobs do not immediately shut down, but they also don't seem to be connecting. status() shows 0 connections.

Ok, that makes sense, (1) just rules out some weird timing issue where the host socket hasn't been set up yet when the daemon tries to dial in, and that's not the case.

[Aariq]

I think i probably have to provide an IP address that is not the first one output by ifconfig for the url argument, but we are still figuring out which IP address is the right one

[shikokuchuo]

I just got some more information from my HPC professionals. Since upgrading to RHEL9, they've had a similar issue with MPI where they have to specify which network card to use because the default one is only for administrative use.

crew uses the getip package to try and work out the correct IP address - but it could be using the wrong interface in your case. It's likely your batch nodes have stricter permissions than other interactive nodes as you say you can use launch_remote() correctly - that shows it's not a mirai connection issue per se.

[Aariq]

Sweet, I was able to get this to work by supplying the correct IP address. Thank you!

It's the IP address returned by ifconfig enp1s0f1np1 | grep inet

[shikokuchuo]

Glad I could help, and that it's working for you now!

[Aariq]

Update: if I run the daemons() call above in one interactive session, then copy the R code output of launch_remote() and run it in a second interactive session, it successfully connects.