sash process OOM / timeout for hypermutated samples

[qclayssen]

In sash runs, SNV process involving PCGR (annotate, report) and SigRap/MUTPAT fail for hypermutated samples due to OOM or time-outs on current process config. Regular samples are fine.

A quick fix is to add:

errorStrategy = 'retry'
maxRetries    = x

which restart the process with x * memory and x * cpu

---

Another possibility is to select an adaptive config file based on DRAGEN results (e.g: variant load, contamination). That would require reading those metrics upstream in Step Functions and propagating them to the execution of the workflow.
@alexiswl is this have been considered ?

[alexiswl]

@alexiswl is this have been considered ?

Definitely hasn't, I think the former sounds like a quick fix?

Also I assume that the logic on memory requirements proportional to sample input is a pretty complex formula and may require a lot of tinkering.

Is the retry approach something you're okay to develop @qclayssen?

[qclayssen]

Retry is already there; I’ll just turn it on in the config, I will do it.

For adaptive memory, we wouldn't need a proportional formula. We could use a discrete selection from DRAGEN metrics.

e.g

if dragen_num_variants > 500_000:
   payload  "tag" = "hypermutated"

nextflow run ... -profile docker,[tag]

with resource config adapted for larger/hypermutated sample

[alexiswl]

For the adaptive config, if you're happy with a binary 'hypermutated' / 'nonhypermutated' tags for an analysis, would you mind adding this in as a feature request into https://github.com/orcabus/service-dragen-wgts-dna-pipeline-manager, currently I'm supporting 'pre' tags, as expected coverage, and expected duplication rate and want to support 'post' tags as well, where post-analysis we add in the tags with the actual calculated coverage or actual duplication rate. We could also add in metrics such as the number of snv called or sv or cnv variants called too once the analysis is complete and oncoanalyser / sash could use those metrics to infer the memory / cpu sizes that they should use for tasks