csv file split into channels from file online

[hkaspersen]

I have uploaded a csv file to github, and want to use it as input for a pipeline test.
The CSV contains a list of URLs to each fasta file I wanted to include (also uploaded to github). The file looks like this:

id,path
isolate1,https://raw.githubusercontent.com/path/to/isolate1.fasta
isolate2,https://raw.githubusercontent.com/path/to/isolate2.fasta

I am trying to make Nextflow create one channel that contains the actual fasta files listed inside the csv file, but I cannot seem to make it work.
I have tried the following:

params.input="https://url/to/csv_file.csv"

Channel
    .fromPath(params.input)
    .splitCsv(header:true, sep:",")
    .collect()
    .set { new_channel }

When running this, the output of the process using the channel only creates a symlink named input.1 which contains the following:

[sample:isolate1, path:https://raw.githubusercontent.com/path/to/isolate1.fasta]

Instead of the actual fasta files.
Any suggestions on how to make this work is appreciated!

[jorgeaguileraseqera]

Hi @hkaspersen

What I don't fully understand is the output you obtain. Is it only one element? I've tried with my csv and collect returns to me all elements

[hkaspersen]

Hi @jorgeaguileraseqera
Sorry for not being clear here.
The process that is using this channel is running an analysis that includes all the fasta files listed in the .csv file. Thus, I need a channel that can download each and every fasta file listed so I can use them in the process.
The output listed there is found in the work directory of the process. It doesn't seem like it is taking all of the entries in the csv file into account somehow, and instead of downloading the files through the URLs in the csv, it symlinks to a file that contains the URL itself.
When I download the csv file outside of nextflow I get the correctly formatted file, so there is nothing inherently wrong with the file itself.

[jorgeaguileraseqera]

I have for example this data.csv

id,path
isolate1,https://www.nextflow.io/docs/latest/process.html
isolate2,https://www.nextflow.io/docs/latest/channel.html`

and this pipeline

process test{
  input: 
    tuple val(id), path('data')   
  output:
    stdout
  script:
    """
    echo $id
    cat $data
    """  
}

workflow {

  chn = Channel
      .fromPath( "data.csv" )
      .splitCsv(sep:',', header:true)
   
  test( chn ) | view
}

process test will receive 1 tuple with the id and the path. We define that the first item in the tuple is a string and the second is a path so nextflow will detect it's a remote file and we can use it (for example cat the file).

This process will be called 2 times (one per row in the csv)

when I run this pipeline I can see these files are displayed in the console.

[hkaspersen]

Thanks for your quick reply! Will I then be able to put each fasta file to a channel through stdout then?

process test{
  input: 
    tuple val(id), path('data')   
  output:
    stdout, emit: fasta_ch
  script:
    """
    echo $id
    cat $data
    """  
}

[jorgeaguileraseqera]

not necessarily, my pipeline was an example of how a process receives tuples from a csv and how to convert some of the values to a path

I mean, in your test process you can access the file (I called it data but it can be whatever you want) so you can operate over it

[jemunro]

I think what you want is something like this:

params.input="https://url/to/csv_file.csv"

fasta_channel = Channel
    .fromPath(params.input)
    .splitCsv(header:true, sep:",")
    .map { file(it.path) }

or if you want to keep the IDs

...
   .map { [it.id, file(it.path)] }

[hkaspersen]

Thank you for this! This is exactly what I needed! Simple, yet brilliant.