How could I use SGE for job submission?

[QuantumChemist]

I know jf remote has interfaces for slurm and PBS submission, but @nfragapane is required to use SGE.

We try to use the qverbatim option like

resources = {"qverbatim": "#$ -cwd\n#$ -pe smp 128\n#$ -N Autoplex_jf_test\n#$ -o $JOB_ID.log\n#$ -e $JOB_ID.err\n#$ -P cpu\n#$ -l s_rt=05:00:00"}

but this yields the following submission script:

#!/bin/bash

#PBS -N add
#PBS -o /user_path/autoplex_jobflow_outputs/job_dir/queue.out
#PBS -e /user/autoplex_jobflow_outputs/job_dir/queue.err
#$ -cwd
#$ -pe smp 128
#$ -N VASP-Job
#$ -o $JOB_ID.log
#$ -e $JOB_ID.err
#$ -P cpu
#$ -l s_rt=05:00:00
cd /user_path/autoplex_jobflow_outputs/job_dir
module load mpi/openmpi-x86_64
source activate autoplex_jobflow
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
export OMP_NUM_THREADS=1

jf -fe execution run /user_path/autoplex_jobflow_outputs/job_dir

But we would like to get rid of the three "#PBS" lines as we think this is what is causing the jobs to fail.
Is there a way to achieve this with qverbatim or might it also be an option if e.g. I and @nfragapane implement an SGE interface?

Thank you a lot in advance! 😃

[davidwaroquiers]

Hi @QuantumChemist

Thanks for raising this issue. Not totally sure the #PBS lines are causing the jobs to fail. jobflow-remote uses qtoolkit under the hood for creating the script and submitting it. I'd guess it could be more likely that the submission and status commands are the problem. Maybe could you try submitting this job script "manually" and see if sge rejects it ? If it does not that means the #PBS lines are not the reason.

I don't think there is a direct way to use qverbatim to do what you want. There may be a template by changing the template but not sure if this could be done cleanly. Implementing the SGE interface in qtoolkit would indeed probably be the best option. I don't think it would be too difficult to do. (or maybe we could add a generic scheduler but I think it would be quite complex 1/ to develop and 2/ to use, and there are not so many scheduler types)

You can look within qtoolkit here: https://github.com/Matgenix/qtoolkit/tree/develop/src/qtoolkit/io where there are the slurm, pbs and shell (i.e. running a process "locally", e.g. on the "frontend" of a host) implementations. I would suggest to also take some inspiration from aiida and/or fireworks (and/or others ?) sge implementations. The reason why we developed qtoolkit is to have such a thing extracted out from those larger packages. Ultimately, it would be nice that aiida, jobflow-remote, fireworks, and all other workflow managers use qtoolkit (well it could be another scheduler-dedicated package but there was none that we could find when we started qtoolkit + jobflow-remote).

Happy to help and review on qtoolkit if you have any questions. (note @gpetretto and I are on holidays until 19 August though).

Best

[QuantumChemist]

Hi @davidwaroquiers ,

thank you for the swift reply! :)

Usually @nfragapane uses a submission script that resembles the one I posted (just without the PBS lines), but we will doublecheck and try to submit this jf remote specific one manually.

I think I can manage an SGE interface implementation in qtoolkit by at least playing copycat with the existing slurm/PBS interfaces and aiida etc. :D and will check the funcitonality together with @nfragapane then!

Happy holidays to both of you then!

[ml-evs]

I'm still around for August and happy to review any PRs, feel free to ping me when its ready!

[QuantumChemist]

I'm still around for August and happy to review any PRs, feel free to ping me when its ready!

Nice! Thank you :)