Do we need yet another queuing system interface?

[jan-janssen]

While the materials project, Aiida and pyiron all started from different directions and developed different approaches to high throughput atomistic simulation in materials science, I wonder if we could agree on a shared standard to interface with the queuing system. Especially, as we already all converged to using fabric / paramiko for handling the SSH connection, two factor authentication and so on.

In analogy to using conda-forge for package distribution and optimade as a central interface for accessing atomistic databases, I think agreeing on a joined package for queuing system submission would allow users to test different workflow frameworks more easily and reduce the workload the developers.

For pyiron the queuing system interface is available in the pysqa package. To simplify the configuration for the users pysqa uses jinja2 templates for converting the submission script templates provided by the computing centres to templates which can be used by pysqa - SLURM example.

I am aware that the ExaWorks project attempted the same with their development of the psij-python package still I feel that if we could find a shared bases between the materials project, Aiida and pyiron that could already be a step in the right direction.

[davidwaroquiers]

Hello @jan-janssen

Thanks for opening this issue for discussion! I fully agree that it would be very beneficial to converge on a common interface for queuing systems. This is what we actually tried to start (alone, I must admit) with qtoolkit. As you mention, MP, Aiida and pyiron (and others) all have their own approaches to "interact" with the queueing systems but this was usually buried inside the codes (e.g. within fireworks code base, within aiida-core, ...). We tried to build qtoolkit as a "standalone" queue interface, with no (or at least no required) dependencies (this is, I think a prerequisite for such a low-level tool, to make sure no "cross-dependency-hell" occurs) and we would definitely be happy to discuss and try to converge towards a common solution (note that we developed qtoolkit for use in jobflow-remote, another package that deals with execution of jobflow workflows from the Materials Project stack).

Pinging @giovannipizzi @utf @computron @gpetretto @gmrigna for thoughts/follow-up on this.

[gpetretto]

Hi @jan-janssen,

thanks for raising this issue. I definitely agree with your point of view. When we started this project we were rather tight on times, but we still spent some time checking the available packages. As David said, in most cases the actual functionalities are buried inside large packages with many dependencies. Also, most of them depend on some kind of "queue manager", that mostly fits the use case of the larger package. What I believe is the key point is having a package where the basic functionalities are directly and clearly exposed and not hidden in some queue manager.
With basic functionalities I mean:

• templates for generating submission scripts
• functions to generate the common commands (e.g. get list of jobs, submit a new job)
• function to parse the outputs of the commands (e.g. extract the list of jobs) and understanding if the response contains an error.

In general, I expect that it will be much more difficult to have a "one size fits all" queue manager that could be used by everybody. Each workflow manager has different requirements depending on how it works, but sharing those common funcitonalities would be already a nice step forward. I am definitely willing to work on a shared solution for exposing the basic functionalities and reduce code duplication.

[jan-janssen]

Obviously I am a bit biased, as those are exactly the challenges I tried to address with pysqa when I created the standalone package back in 2019. Still I am happy to adopt any queuing system adapter as long as it provides an easy way for the users of pyiron to configure the queuing system with the information which is typically provided by the computing centres. To keep this discussion productive, I am going to link the corresponding parts of pysqa to enable a quick comparison.

With basic functionalities I mean:

• templates for generating submission scripts

In pysqa we use jinja2 templates to convert the queuing system templates provided by the computing centre to templates which can be used inside pysqa. To me this seems like an easy way for the user to replace their existing scripts and convert them. https://pysqa.readthedocs.io/en/latest/queue.html#slurm

A generic set of the variables is shared between all queuing systems: https://pysqa.readthedocs.io/en/latest/queue.html#queuing-systems

• job_name - the name of the calculation which appears on the queuing system
• working_directory - the directory on the file system the calculation is executed in
• cores - the number of cores used for the calculation
• memory_max - the amount of memory requested for the total calculation
• run_time_max - the run time requested for a given calculation - typically in seconds
• command - the command which is executed on the queuing system

Still the users can also use kwargs and add their own custom variables. At the moment the assignment of gpus is handled as such a custom variable.

• functions to generate the common commands (e.g. get list of jobs, submit a new job)

For the different queuing systems pysqa provides wrappers https://github.com/pyiron/pysqa/tree/main/pysqa/wrapper . These define a set of abstract command functions and return the corresponding shell commands as a list of strings, currently we implement the following commands:

• submit_job_command()
• delete_job_command()
• enable_reservation_command()
• get_queue_status_command()

• function to parse the outputs of the commands (e.g. extract the list of jobs) and understanding if the response contains an error.

For this pysqa defines:

• get_job_id_from_output(queue_submit_output) - a function which returns the queuing system identifier after the successful submission to the queuing system.
• convert_queue_status(queue_status_output) - a function which converts the output of get_queue_status_command() to a pandas DataFrame.

but sharing those common funcitonalities would be already a nice step forward. I am definitely willing to work on a shared solution for exposing the basic functionalities and reduce code duplication.

Yes, I completely agree. I am sure that some specific requirements are going to remain but if we can already agree on a shared library which covers part of the challenges, that is already a step in the right direction.

[ltalirz]

Hey guys, I was pointed to this thread by @jan-janssen

Just two small thoughts from my side:

1. throwing the myqueue project from the ASE folks in the hat as well (I haven't used it myself; just looks like its purpose is similar)

2. In order to improve security, HPC centers are slowly moving from SSH + command line client interfaces to REST APIs for job submission and file transfer. This is also how cloud native compute backends tend to work.

Point 2 brings two important changes: (a) around authentication (moving from from time-unlimited SSH keys to time-limited tokens), and (b) restrictions around what users can do; in particular they won't allow the user to run arbitrary commands on the system (see e.g. https://firecrest-api.cscs.ch/ for an idea of what such an API can look like; or here is the slurm rest api https://slurm.schedmd.com/rest_api.html).

It can be a valid choice to ignore these APIs for now - I guess it will take a while until a standard emerges - but if possible, try to anticipate some of these changes in the design of the package, e.g. by making it possible to implement other forms of authentication & check whether the functionality provided by the package could be mapped to the REST endpoints in these APIs.

In the end these can be small things - e.g. AiiDA assumed it could do posix file system operations on the remote storage, which does not map to these APIs.

[jan-janssen]

1. throwing the myqueue project from the ASE folks in the hat as well (I haven't used it myself; just looks like its purpose is similar)

To just comment on the myqueue part and why we do not use it in pyiron:

• Building up a database to track which jobs are currently running on the queuing system is an anti-pattern to me. The queuing system already has a database and duplicating it only causes more trouble.
• The way resources are configured in myqueue is very different from the submission scripts provided by the computing center as myqueue only enables the specification of resources via nodenames (SLURM --partitions).

The pysqa package addressed these challenges by directly interfacing with the queuing system without the need of a separate database - in pyiron we build a database on top but that one also tracks the relationship of the individual job objects and so on, so to me this is not part of a queuing system adapter and we introduced the jinja2 templates for the submission scripts to prevent our users from having to specify their resources in a complicated way.

Still I agree that if we build a minimalistic queuing system adapter which just provides the commands for different queuing systems, then such a tool might also be relevant for the developers of myqueue and it would be possible to integrate such a tool in myqueue.

[davidwaroquiers]

Hi @ltalirz,

Thanks for your thoughts! Regarding REST API's, definitely this is something to be kept in mind and we actually considered it initally. We started "without" it but not preventing from moving to it. Our reasoning was that in any case, currently most computing centers are still providing either only "standard" command line or both command line + rest api.

[giovannipizzi]

Hi all, very interesting thread. In AiiDA, while the code for scheduler management is part of the main aiida package, it is actually by design decoupled in a submodule (aiida.schedulers) that does not have any AiiDA dependency, so it could be easily factored out in a separate module, or combined in the module being discussed here. The logic is quite similar to the one described above, with a scheduler-agnostic template to define a new job, a scheduler agnostic class to store information about each job in the queue and generic methods in the base class to get the command to list, kill, ... jobs, and parse the respective outputs.
Actual implementations are subclasses, implemented via a plugin system (via entry points, so they can be part of a different python package and AiiDA will discover those plugins if they are installed via pip). Core plugins (SLURM, Torque, ...) are here and more are available on the plugin registry of AiiDA.

The only addition is that it supports tunnelling those commands via the transport interface (also part of AiiDA in aiida.transport, also independent of the rest of AiiDA), i.e. it's agnostic to whether those commands are run locally, over ssh, or via other connection approaches (also those extensiblee via plugins).

So, converging on a single package to manage the scheduler wouldn't be a huge amount of work (even if as @gpetretto mentions, probably when one goes to see the details, things become a bit more complex).

However, an ongoing project in AiiDA is indeed to extend the scheduler part to support completely different approaches to deal with REST-API-based schedulers, in particular FirecREST already mentioned by @ltalirz.
This requires quite some thinking of the correct level of abstraction as already mentioned, and we've already done some refactoring of the code to e.g. not assume anymore that to get the list of jobs there is a single command you can run on a command line.
And for us this is a requirement, because going via the REST API will become soon a mandatory requirement in our centers, and something we are actively working to support.

Therefore, I'd be hesitant to invest energy in a new package if this requirement is not included already in the design, since it will not bring additional benefit to most of us who already have the code to deal with schedulers. If we instead address this issue from the beginning, then the new package becomes very powerful because in addition to providing existing functionality independent of the underlying engine, it's also future proof.

Pinging here @khsrali who's actively working on the support for FirecREST in AiiDA, and also @GeigerJ2 @sphuber @mbercx for info

[khsrali]

Hi there,
Very interesting discussion.

yes I agree with @giovannipizzi. The way AiiDA is designed, makes it possible to converge to other packages for scheduler & transport relatively easy.

In general, is a great idea to have a standard package that can deal with REST-API-based schedulers.
However, the problem right now is, some of those schedulers although functional, are not yet fully production ready in terms of performance.
I agree it makes sense that everybody use a standard package, although sounds like producing such package may still inevitably wait for the above reason.

[davidwaroquiers]

Thanks all for your comments, suggestions and insights. It seems to me there is enough attraction and interest to try to converge to a common package/api for both cmdline based and restapi based schedulers (possibly transport ? even if I would probably see that as a second or separate topic). I propose we have a first meeting to discuss about needs, constraints, broad view on this topic. Would that be ok for everyone ? If so, I can send a doodle to choose a date.