fix(new delivery service calls in CLI)

[ChrOertlin]

Description

Implements the new delivery service in the CLI calls

1. 'cg deliver analysis' changed to 'cg deliver case' -> now only delivers files on a case basis to the customer inbox
2. 'cg deliver ticket' changed behaviour, does no longer RSYNC files automatically.
3. Removed 'cg deliver concatenate' as concatenation is done automatically now where it is required.

Motivation:
Previously cg deliver analysis had functionality for both ticket and case delivery in one even though a cg deliver ticket cli command exists.
Streamlined the functionaliy of the cg deliver case/ticket, now both function just deliver files to the customer inbox on the HPC. A separate Rsync call is now required.

Added

Changed

Fixed

How to prepare for test

• Ssh to relevant server (depending on type of change)
• Use stage: us
• Paxa the environment: paxa
• Install on stage (example for Hasta):

  bash /home/proj/production/servers/resources/hasta.scilifelab.se/update-tool-stage.sh -e S_cg -t cg -b [THIS-BRANCH-NAME] -a

How to test

• Do ...

Expected test outcome

• Check that ...
• Take a screenshot and attach or copy/paste the output.

Review

• Tests executed by
• "Merge and deploy" approved by
  Thanks for filling in who performed the code review and the test!

This version is a

• MAJOR - when you make incompatible API changes
• MINOR - when you add functionality in a backwards compatible manner
• PATCH - when you make backwards compatible bug fixes or documentation/instructions

Implementation Plan

• Document in ...
• Deploy this branch on ...
• Inform to ...

[karlnyr]

In production we need to have one command which does the full delivery, meaning both to the customer inbox on hasta and then from hasta to Caesar. Having this command would reduce the possibility that one would forget to run the rsync command.

Otherwise, I think that this logic looks a lot cleaner and easier to understand. However, I was wondering about the removal of the --ignore-missing-bundles and --force-all flag. Will we now always ignore missing bundles? And will we always transfer all data available?

The merge of this PR would require that you change the following document to be up to date: https://atlas.scilifelab.se/production/data_analysis/workflows/raw_data_delivery/

[ChrOertlin]

In production we need to have one command which does the full delivery, meaning both to the customer inbox on hasta and then from hasta to Caesar. Having this command would reduce the possibility that one would forget to run the rsync command.

Alright, then I will add the rsync to both.

Otherwise, I think that this logic looks a lot cleaner and easier to understand. However, I was wondering about the removal of the --ignore-missing-bundles and --force-all flag. Will we now always ignore missing bundles? And will we always transfer all data available?

It fetches what is available and delivers it. If a bundle is missing it will be skipped by default.
The command will overwrite existing data in the HPC on hasta and with the rsync change will start the sync

The merge of this PR would require that you change the following document to be up to date: https://atlas.scilifelab.se/production/data_analysis/workflows/raw_data_delivery/

[Vince-janv]

Looks really nice! Let's make sure all of the weird logic is covered though. I'll post a little list

[Vince-janv]

Did not mean to request changes 😅

[Vince-janv]

It fetches what is available and delivers it. If a bundle is missing it will be skipped by default. The command will overwrite existing data in the HPC on hasta and with the rsync change will start the sync

I really like the logic described above. I think simplicity is something we should really strive for here. @karlnyr Do you see any potential problems with it?

[Vince-janv]

Here's a list of strange things we need to include from the top of my head.

Concatenation of microSALT cases

• Cases with data-analysis microSALT should have their FASTQ file concatenated when delivered with the delivery type FASTQ (and FASTQ_QC, FASTQ_QC+ANALYSIS)

"Missing bundles"

• Cases of type FASTQ, microSALT, and Mutant should not raise an exception if a one of its samples is missing a bundle

CLI commands

• A CLI command exists to deliver (just link to cust inbox on hasta) a whole ticket just a case.
  • It should take a delivery type as input as well (ie one should be able to specify delivery for a case that has data_delivery=scout)
• A CLI command exists that which takes a ticket and, then links the correct files and starts an rsync job. If this ticket is a covid ticket extra logic needs to be applied in the rsync part (should be able to just swap out the first part of this command)

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[ChrOertlin]

Merging into dev branch - copied the required tests into the dev brancnh as well.