Submitit Plugin w/ Hydra failing due to no localhost connection

[tyleryzhu]

I'm trying to run the submitit plugin with hydra, which works without submitit, i.e. locally with torchrun. However, I'm running into this tricky error that I can't understand, potentially being a problem with either multi-run or otherwise. My slurm configuration works perfectly fine, it is just combining it with hydra that has not worked yet. Any pointers? Otherwise any suggestions for good workarounds that can keep hydra for my main.py but launch submitit otherwise?

submitit WARNING (2023-10-25 05:57:37,983) - Call #6 - Bypassing sacct error Command '['sacct', '-o', 'JobID,State,NodeList', '--parsable2', '-j', '45469']' returned non-zero exit status 1., status may be inaccurate.
[2023-10-25 05:57:37,983][HYDRA] Call #6 - Bypassing sacct error Command '['sacct', '-o', 'JobID,State,NodeList', '--parsable2', '-j', '45469']' returned non-zero exit status 1., status may be inaccurate.
sacct: error: slurm_persist_conn_open_without_init: failed to open persistent connection to localhost:6819: Connection refused
sacct: error: slurmdbd: Sending PersistInit msg: Connection refused
sacct: error: Problem talking to the database: Connection refused

[odelalleau]

You could inspect the SLURM scripts generated by submitit to check if all looks good. They should be somewhere inside your multirun dir (check with ls -a to see the .hidden folders).

[tyleryzhu]

Here's what my hydra config looks like to generate the multirun (simplified obviously):

defaults:
  - override hydra/launcher: submitit_slurm

hydra:
  run: 
    dir: ${log_dir}
  sweep:
    dir: ${hydra.run.dir}
    subdir: ${hydra.job.num}
  launcher:
    submitit_folder: ${hydra.sweep.dir}/.submitit/%j 
    name: ${name}
    tasks_per_node: ${hydra.launcher.gpus_per_node} 
    timeout_min: 4320 
    cpus_per_task: 4 
    gpus_per_node: 2
    gres: "gpu:${hydra.launcher.gpus_per_node}"
    tasks_per_node: ${hydra.launcher.gpus_per_node}
    nodes: 1
    _target_: hydra_plugins.hydra_submitit_launcher.submitit_launcher.SlurmLauncher
    qos: low
    signal_delay_s: 120

Here's what the output generated submitit with hydra looks like:

#!/bin/bash

# Parameters
#SBATCH --cpus-per-task=4
#SBATCH --error=/home/tyler/project/logs/.submitit/%j/%j_0_log.err
#SBATCH --gpus-per-node=2
#SBATCH --gres=gpu:2
#SBATCH --job-name=train_model
#SBATCH --nodelist=em4
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=2
#SBATCH --open-mode=append
#SBATCH --output=/home/tyler/project/logs/.submitit/%j/%j_0_log.out
#SBATCH --qos=low
#SBATCH --signal=USR2@120
#SBATCH --time=4320
#SBATCH --wckey=submitit

# command
export SUBMITIT_EXECUTOR=slurm
srun --unbuffered --output /home/tyler/project/logs/.submitit/%j/%j_%t_log.out --error /home/tyler/project/logs/.submitit/%j/%j_%t_log.err /home/tyler/miniconda3/envs/project/bin/python -u -m submitit.core._submit /home/tyler/project/logs/.submitit/%j

It doesn't look so different from the other logs, but also I don't see any reference of my main script here. Is that intentional?