Add seff to Toil Slurm?

[adamnovak]

The seff command can get statistics about how much CPU and memory a job actually used.

It might be useful for Toil to expose this information in its logs somehow when running on Slurm, either to diagnose OOM or to diagnose excessively large job resource allocations.

┆Issue is synchronized with this Jira Story
┆Epic: Improve debugging experience
┆Issue Number: TOIL-1349

[DailyDreaming]

Toil could maybe have an alert system when resource usage is too high maybe? Making sure that the spurious to useful alerts wouldn't be too high, though this may be difficult? We should discuss this later, but it might be low priority for now.

[mr-c]

Yes, this would be good to have; and report as part of RO-Crate WorkflowRun provenance

[adamnovak]

This might also be useful for Dockstore as part of a workflow analytics report-back feature/a bigger system.

[unito-bot]

➤ Adam Novak commented:

And it would be good for Toil to warn you if your jobs are over-provisioned and wasting cluster CPU and memory, rather than you having to manually guess which Slurm job is which and ask Slurm about them all.

[unito-bot]

➤ Adam Novak commented:

We should put this into toil stats or figure out if it is redundant with stuff toil stats already does.