Empty fastq files in delivery for microsalt cases

[seallard]

Description

At least one microsalt case has been observed to contain two empty fastq files (besides the fastq files containing the concatenated data) in the customers delivery directory on Hasta.

BLAZA-9789:
total 648486
-rw-rw----  1 hiseq.clinical  cust015          0 Feb 26 10:03 240219_BLAZA-9789_1.fastq.gz
-rw-rw----  1 hiseq.clinical  cust015          0 Feb 26 10:03 240219_BLAZA-9789_2.fastq.gz
-rw-rw----  1 hiseq.clinical  cust015  339066091 Feb 26 10:03 BLAZA-9789_R1.fastq.gz
-rw-rw----  1 hiseq.clinical  cust015  324937459 Feb 26 10:03 BLAZA-9789_R2.fastq.gz

The reason the files are not removed is that they do not follow the expected naming pattern for sample fastq files.

Suggested solution

Determine where the empty files are created.
Ensure they are not created, alternatively remove them before delivery.

This can be closed when

The empty fastq files are not present.

[seallard]

The empty files are not present in the bundle for the case in housekeeper. So they are not generated by the microsalt pipeline.

Looking at the names, the empty files were generated by the cg deliver ticket command. So the issue is that:

1. Automation delivered the files, concatenated and removed them.
2. The manual deliver ticket command was run separately which created the empty files (since automation already concatenated and removed the other ones).

[seallard]

Closing this for now, re-open if issue reoccurs.

[eliottBo]

I observed this issue again (2024-03-11) when using cg deliver ticket -d fastq -t <ticket_id> --force-all. I removed the additional empty files with find . -type d -exec sh -c 'cd "{}" && rm -f <pattern>*' \; command in the directory.

I used this command because a sample in a microsalt case did not receive enough raw reads but the QCs in the report were ok and I decided to deliver as it was. After setting as completed in TB, I removed the ticket folder on caesar and hasta and used the cg deliver command with --force-all flag to include the sample without enough raw reads.

[seallard]

I'm pretty sure this means the sample had already been concatenated and delivered.
I think the QC for microsalt allows delivering failing samples given that the case still passes QC. Is that something you want to change?

See documentation here #2754 (comment)

I'll patch the concatenation logic so that it does not generate the empty files, but you should be aware that the failed sample most likely already had been delivered.

[Vince-janv]

Does not seem solved. Could replicate the issue.
If cg.services.fastq_concatenation_service.utils.concatenate gets an empty list as input it will create an empty file

[Vince-janv]

Should be solved by the merge of the new delivery flow