Skip process if downstream processes are done

[murphycj]

I've been reading though the docs and past discussions, but I cannot understand how to make my use case work.

To give an example, say I have two processes with the following DAG:

A -> B

Process A downloads a giant file, and B compute some summary statistics on it and stores the result in a storeDir. The files downloaded in A are so large I need to delete them once the workflow is done. But the issue is that over time I need to run new samples through the workflow, but I don't want A to re-download the files I deleted... because if it does I'll run out of disk space.

Is there a way to have nextflow not re-run A if output files for B exist? This is assuming the outputs for past runs of A have been deleted. Furthermore, I want to be able to extend the workflow by adding a process C after B, so I'll need it to run the workflow again for all past samples as well (but again, without running A again).

[bentsherman]

I think this example is a use case for the solution described here: #452 (comment)

Basically you need a mechanism to delete intermediate files once they are no longer needed, but also not re-compute them if their downstream outputs already exist.

[murphycj]

Thanks for the reply and link! Yes that would solve my issue. Such a feature would be immensely useful. A necessary feature for any workflow system (in my opinion) :)