SBATCH ERROR: File "disbatch" could not be opened from the directory

[andrewkhardy]

I am attempting to run disBatch on Niagara. I can successfully install disBatch following the instructions

python -m venv venv
. venv/bin/activate
pip install disbatch

However if I try to run a script which contains the command

julia --project=../Project.toml --heap-size-hint=4G tJ.jl &> tJ.log

which I run without SLURM using the command

disbatch disbatch_Examples

I get the error

Cannot determine batch execution environment.
Exception in thread Driver:
Traceback (most recent call last):
File "/cvmfs/soft.computecanada.ca/gentoo/2023/x86-64-v3/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner

and when I try to run on SLURM via the command

sbatch --account=rrg-aparamek disbatch disbatch_Examples

I get the error

SBATCH ERROR: File "disbatch" could not be opened from the directory "/gpfs/fs1/home/a/aparamek/andykh/t_J.jl/Examples" on host "nia-login05.scinet.local"

I should note that I did not make venv in this directory, but if I run the same command from my home directory where venv is located, then I get the same error.

I'm posting this under Discussions rather than Issues because it's likely that I just need to setup something, but I use disBatch on Popeye with absolutely no trouble, so I was not expecting much resistance.

[lgarrison]

disbatch disbatch_Examples

Cannot determine batch execution environment.

I think this error is normal; to run without Slurm, you can use something like

disbatch -s localhost:4 disbatch_Examples

This will execute 4 tasks at once on the node where you're running the command.

SBATCH ERROR: File "disbatch" could not be opened from the directory "/gpfs/fs1/home/a/aparamek/andykh/t_J.jl/Examples" on host "nia-login05.scinet.local"

Hmm, it sounds like Slurm can't find the disbatch executable. Just to double check, is the venv activated? What does which disbatch return? How about sbatch --wrap 'which disbatch'?

I should note that I did not make venv in this directory, but if I run the same command from my home directory where venv is located, then I get the same error.

The location of the venv relative to the working directory shouldn't matter once the venv is activated.

[andrewkhardy]

1. If I follow Quickstart instructions correctly, the jobs run on the login node using localhost.

2. which disbatch specifies the correct directory of the venv, /gpfs/fs1/home/a/aparamek/andykh/venv/bin/disbatch.

sbatch however (obviously in hindsight) does not have venv activated and so it does not load. I would normally correct this by adding . venv/bin/activate to the SLURM script, but I need SLURM to load it before using disbatch to do say source SetupEnv which is where modules are loaded. I could not get it to work with a #DISBATCH PREFIX ( either?

[lgarrison]

I would just recommend activating your venv (or sourcing your environment script) before calling sbatch. That will put disBatch on the PATH. You can also write a batch script where you source the environment and then call disBatch, but that's probably overkill unless you're doing more than just calling disBatch in the batch script.

The other way to solve this is with pipx install disbatch or uv tool install disbatch. Either of these will create a dedicated venv just for disBatch and transparently use it. You can read more about this here: https://github.com/flatironinstitute/disBatch#direct-invocation-with-pipx-or-uvx

[andrewkhardy]

The issue is that on Niagara, what environment I have activated on the login node is not activated on the compute node where SLURM executes the command. When I get the error, I have venv activated (hence why which disbatch works), but the SLURM does not seem aware of that.

[lgarrison]

Hmm, I thought Slurm usually preserved the calling environment (sbatch --export=ALL is the default). Maybe Niagara is different somehow. You could try --export=ALL (or --export=PATH), or specify the absolute path to disbatch (e.g. sbatch $(which disbatch) tasks), or use a wrapper batch script.