Question: What does input of "path '*'" do in script6.nf?

[jgbaum]

I've been working through the tutorials here that make use of this repo and I've followed along to this point, but am trying to wrap my head around how the following line in the multiqc process is working:

https://github.com/nextflow-io/training/blob/master/nf-training/script6.nf#L72

If you could help to explain and/or point me to documentation, that would be much appreciated. Thanks!

[mribeirodantas]

The input block has the following format:

input:
<input qualifier> <input name>

The input qualifier specifies what type of input this process expects. The path qualifier says it's either a single file, multiple files, single folder or multiple folders. For teh path qualifier, the input name can either be a variable or the name of the file/folder. If the MULTIQC expected a single file named my_input.csv, we'd have:

input:
path 'my_input.csv'

But if you follow the workflow scope of the script6.nf, you'll see we use the collect channel operator before providing the input to the MULTIQC process, which means that every channel element is a list with multiple files/folders, so even a single MULTIQC task will receive multiple files as instances. These files will be staged in the task folder. So when we say path '*' in the MULTIQC process, we're just saying to use all these files as input to this process.

[jgbaum]

Got it! Things are finally starting to click for me now. What was confusing to me was the asterisk / all files as input, yet they were never directly referenced in the script. Now (I think) I understand that any paths that are specified as input to a process are staged into the working directory for the task. Good stuff!

[mribeirodantas]

It's a bit different.

Everything specified as the output of process A will be staged in the working directory of any instance of a process that receives the output channel of A. So if A outputs 10 files, and we do B(A.out) and C(A.out) (i.e., tell Nextflow that B and C depend on the output of A), these 10 files will be linked to the working directory of processes B and C instances. Then, in the process scope of B and C, you could have only part of these files as inputs, but the other ones (links) will still be there, because they're the output of A and you said B and C receive the output channel of A. You can also apply channel operators to the output of process A so that only part of the outputs will be passed downstream to process B and C.

Is this clear?

[jgbaum]

Thanks, I think that's what I was trying to say but I was not precise. Basically, if a channel contains file paths that are consumed by as input to a process with a 'path' type, the files matching the glob will be staged into the working directory for that task? That would explain why the asterisk of the multiqc process causes all files specified by the quant_ch channel to be staged in that directory. Thanks for helping me work through this!

[mribeirodantas]

Yeah. Now that I read again what I wrote it's a bit confusing. In fewer words, regardless of what files are created during your task execution, you should capture with the output block the files that you'll need in the next process. If the appropriate qualifier is used for the process downstream, the files will be staged (linked) in the task directory of the next process.