SLURM Array jobs

[TheRealLoliges486]

Description

Hi,

Would it be possible to add support for the Job Array feature of SLURM in law?

Motivation

I'm looking to submit a large number of jobs (on the order of a million) to my SLURM schedd. According to best practices and recommendations (e.g., ETH Zurich's guidance), using Job Arrays is the preferred way to handle such high-throughput workloads efficiently. It also helps avoid overloading the scheduler and ensures fair usage alongside other users.

Support for this feature would significantly improve scalability and resource management for such large-scale workflows.

Thanks for considering this!

Cheers,
Nico

[riga]

Hey @TheRealLoliges486 ,

yes, this is indeed possible. A while ago, I introduced the generic concept of grouped jobs, which is interpreted by e.g. the HTCondorJobManager as Cluster jobs, which is htcondor's interpretation of the feature you described for slurm.

I'm afraid it is somewhat involved to add support for that and I'm relatively busy in the next 2-3 weeks, so if you need this feature within that time scale, feel free to give it a try. Otherwise, I would put this on my list for May.