Issue with slurm-multicluster

[Jhaquen]

Hi Row team,

Recently our cluster was split into two networks with a shared login node. Since the split we are unable to use row. According to our HPC support team the issue seems to be in scheduler.rs, where an additional string specifying the cluster (mogonki) seems to mess things up.
This is the trace:

Thank you in advance for your time!

PS: you may notice i request CPU hours on a GPU node, this is because our clusters slurm configuration is using --gres not --gpus-per-task. I fix this by passing --gres as custom flag, is this how it is intended?

[joaander]

Thanks for reporting this. It appears that your cluster's job IDs are strings, not integers. Before I can proceed with a fix, I need to understand how squeue works on your system. Does squeue also display job ids as "51245;mogonki"? Can you get the status of a set of jobs with a command like this?

squeue --jobs 51245;mogonki,51246;mogonki,51247;mogonki -o "%A" --noheader

(substitute the IDs of some jobs that are currently executing).

Please share the output of these commands so that I can confirm they behave as expected. Please also paste text as text formatted as a code block to improve readability and allow text based copy and paste.

PS: you may notice i request CPU hours on a GPU node, this is because our clusters slurm configuration is using --gres not --gpus-per-task. I fix this by passing --gres as custom flag, is this how it is intended?

row is a thin interface between your workflow and the scheduler. You are expected to pass whatever arguments are needed to request the intended resources on your cluster's specific SLURM configuration.

[Jhaquen]

Thank you for your very quick reply! It seems job IDs are integers (or at least do not contain "mogonki")

[sschwa02@mogon-nhr01 ~]$ squeue --jobs 886018;mogonki -o "%A" --noheader

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
            886018    a100dl h5rep2Mn ------ PD       0:00      1 (AssocMaxJobsLimit)
-bash: mogonki: Kommando nicht gefunden.


[sschwa02@mogon-nhr01 ~]$ squeue --jobs 886018 -o "%A" --noheader
886018

[sschwa02@mogon-nhr01 ~]$ squeue --jobs "886018;mogonki" -o "%A" --noheader
886018

[sschwa02@mogon-nhr01 ~]$ squeue --jobs 891762_951;mogonki,891762_959;mogonki,891762_966;mogonki  -o "%A" --noheader
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
        891762_951  parallel D251r001  -----  R    5:16:13     16 cpu[0376-0391]
-bash: mogonki,891762_959: Kommando nicht gefunden.
-bash: mogonki,891762_966: Kommando nicht gefunden.
-bash: mogonki: Kommando nicht gefunden.
[sschwa02@mogon-nhr01 ~]$ squeue --jobs "891762_951;mogonki","891762_959;mogonki","891762_966;mogonki"  -o "%A" --noheader
897171
897156
897164
[sschwa02@mogon-nhr01 ~]$ 

row is a thin interface between your workflow and the scheduler. You are expected to pass whatever arguments are needed to request the intended resources on your cluster's specific SLURM configuration.

Is forking to somehow add --gres via action.resources.gpus_per_process encouraged? Could be a fun project!

[joaander]

Thanks for running the test. It seems that all row needs to do is ignore the output ;mogonki output from sbatch.

Is forking to somehow add --gres via action.resources.gpus_per_process encouraged? Could be a fun project!

You are welcome to fork and change the code however you please. However, it is not necessary.
You can use submit_options to set the gres option that is specific to your cluster. Something like:

[action.submit_options.cluster1]
custom = ["--gres=gpu:1"]

Which I think you are already doing. Does also setting gpus_per_process=1 cause an error?

If that doesn't work, I can look into a way to customize the --gpus-per-task string in clusters.toml.

[Jhaquen]

It seems that all row needs to do is ignore the output ;mogonki output from sbatch.

can you estimate how long it will take to make those changes? Again, thank you!

Does also setting gpus_per_process=1 cause an error?

yes it causes SLURM to exit with a syntax error. I was unsure whether GPU allocation was working correctly since row is looking for CPU nodes, but since this is an issue specific to our clusters config im happy to look into it myself!

[joaander]

Please test the branch fix-sbatch-parsing: #197. It ignores non-numeric characters in sbatch output.

To replace --gpus-per-task with another string, set slurm_gpus_per_task in your clusters.toml file. For example:

[[cluster]]
slurm_gpus_per_task = "--gres=gpu:"
...

https://row--197.org.readthedocs.build/en/197/clusters/cluster.html#slurm_gpus_per_task

[Jhaquen]

yes that fixed it!

[joaander]

Thanks for testing the changes!