Add export managed variants #2750

henrikstranneheim · 2023-12-07T16:01:34Z

Description

Add export managed variants and writes to pipeline case dir.

Added

Scout export manged variants function
CLI cmd to export variants
Tests

Changed

Fixed

How to prepare for test

Ssh to relevant server (depending on type of change)
Use stage: us
Paxa the environment: paxa

Install on stage (example for Hasta):

bash /home/proj/production/servers/resources/hasta.scilifelab.se/update-tool-stage.sh -e S_cg -t cg -b add-managed-variant-scout-export -a

How to test

Do ...

Expected test outcome

Check that ...
Take a screenshot and attach or copy/paste the output.

Review

Tests executed by
"Merge and deploy" approved by
Thanks for filling in who performed the code review and the test!

This version is a

MAJOR - when you make incompatible API changes
MINOR - when you add functionality in a backwards compatible manner
PATCH - when you make backwards compatible bug fixes or documentation/instructions

Implementation Plan

Document in ...
Deploy this branch on ...
Inform to ...

…' into add-managed-variant-scout-export

ChrOertlin

looks good!

cg/apps/scout/scoutapi.py

cg/cli/workflow/raredisease/base.py

cg/meta/workflow/mip_dna.py

cg/meta/workflow/raredisease.py

diitaz93

Great 💯

cg/meta/workflow/raredisease.py

sonarqubecloud · 2023-12-08T14:29:03Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.5% Duplication

henrikstranneheim · 2023-12-08T14:46:46Z

cg workflow mip-dna panel justhusky 👍

henrik.stranneheim@hasta:~] [S_base] 3m34s $ cg workflow mip-dna panel justhusky -d
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Case justhusky exists in Status DB
Running command /home/proj/stage/bin/miniconda3/envs/S_scout/bin/scout --config /home/proj/stage/servers/config/hasta.scilifelab.se/scout-stage.yaml export panel --bed BRAIN Cardiology CILM CH CTD DIAB ENDO EP HEARING HYDRO IBMFS IEM IF mcarta MHT MIT MOVE mtDNA NBS-M NEURODEG NMD OMIM-AUTO PANELAPP-GREEN PEDHEP PID PIDCAD RETINA SKD SOVM STROKE AID --build 37

[henrik.stranneheim@hasta:~] [S_base] 57s $ cg workflow mip-dna panel justhusky
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Case justhusky exists in Status DB
Running command /home/proj/stage/bin/miniconda3/envs/S_scout/bin/scout --config /home/proj/stage/servers/config/hasta.scilifelab.se/scout-stage.yaml export panel --bed BRAIN Cardiology CILM CH CTD DIAB ENDO EP HEARING HYDRO IBMFS IEM IF mcarta MHT MIT MOVE mtDNA NBS-M NEURODEG NMD OMIM-AUTO PANELAPP-GREEN PEDHEP PID PIDCAD RETINA SKD SOVM STROKE AID --build 37
head /hom	[henrik.stranneheim@hasta:~] [S_base] 52s $ head /home/proj/stage/rare-disease/cases/justhusky/gene_panels.bed
##genome_build=37
##gene_panel=BRAIN,version=1.0,updated_at=2020-02-14,display_name=BRAIN
##gene_panel=Cardiology,version=1.0,updated_at=2020-01-21,display_name=Cardiology
##gene_panel=CILM,version=59.0,updated_at=2018-03-22,display_name=Ciliopathy
##gene_panel=CTD,version=55.0,updated_at=2021-12-13,display_name=Connective Tissue Diseases
##gene_panel=DIAB,version=17.0,updated_at=2023-03-20,display_name=DIAB
##gene_panel=ENDO,version=12.0,updated_at=2018-02-05,display_name=Endocrinology
##gene_panel=EP,version=15.0,updated_at=2021-12-10,display_name=Epilepsy
##gene_panel=HYDRO,version=4.0,updated_at=2018-01-19,display_name=Hydropspanel

cg workflow mip-dna managed-variants justhusky 👍

[henrik.stranneheim@hasta:~] [S_base] 26s 2 $ cg workflow mip-dna --help
Usage: cg workflow mip-dna [OPTIONS] COMMAND [ARGS]...

  Rare disease DNA workflow

Options:
  --help  Show this message and exit.

Commands:
  config-case                Generate a config for the case id.
  ensure-flow-cells-on-disk  Check if flow cells are on disk for given case.
  link                       Link FASTQ files for all samples in a case.
  managed-variants           Write managed variants file exported from...
  panel                      Write aggregated gene panel file exported...
  resolve-compression        Handles cases where decompression is needed...
  run                        Run the analysis for a case.
  start                      Start full analysis workflow for a case.
  start-available            Start full analysis workflow for all cases...
  store                      Store finished analysis files in Housekeeper.
  store-available            Store bundles for all finished analyses in...
[henrik.stranneheim@hasta:~] [S_base] 25s $ cg workflow mip-dna managed-variants justhusky -d
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Case justhusky exists in Status DB
Running command /home/proj/stage/bin/miniconda3/envs/S_scout/bin/scout --config /home/proj/stage/servers/config/hasta.scilifelab.se/scout-stage.yaml export managed --build 37
##fileformat=VCFv4.2
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">
##fileDate=2023-12-08 15:40:09.807516
##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=TYPE,Number=1,Type=String,Description="Type of variant">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO
1	48696925	.	G	C	.		END=48696925;TYPE=SNV
7	124491972	.	C	A	.		END=124491972;TYPE=SNV
12	103234252	.	T	C	.		END=103234252;TYPE=SNV
14	76548781	.	CTGGACC	G	.		END=76548781;TYPE=INDEL
17	48696925	.	G	T	.		END=48696925;TYPE=SNV

[henrik.stranneheim@hasta:~] [S_base] 47s $ head /home/proj/stage/rare-disease/cases/justhusky/managed_variants.vcf
##fileformat=VCFv4.2
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">
##fileDate=2023-12-08 15:41:25.195938
##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=TYPE,Number=1,Type=String,Description="Type of variant">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO
1	48696925	.	G	C	.		END=48696925;TYPE=SNV
7	124491972	.	C	A	.		END=124491972;TYPE=SNV
12	103234252	.	T	C	.		END=103234252;TYPE=SNV

cg workflow mip-dna start justhusky 👍

[henrik.stranneheim@hasta:~] [S_base] $ cg workflow mip-dna start justhusky --mip-dry-run
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Case justhusky exists in Status DB
Starting full MIP analysis workflow for case justhusky
Flow cell check is not applicable - the case is either down sampled or external.
Found file earlycasualcaiman/2019-05-08/HHT5NDSXX_HapMap_S26_L001_R1_001.fastq.gz
Found file earlycasualcaiman/2019-05-08/HHT5NDSXX_HapMap_S26_L001_R2_001.fastq.gz
Found file slowlycivilbuck/2019-05-08/HHT5NDSXX_HapMap_S27_L001_R1_001.fastq.gz
Found file slowlycivilbuck/2019-05-08/HHT5NDSXX_HapMap_S27_L001_R2_001.fastq.gz
Found file hugelymodelbat/2019-05-08/HHT5NDSXX_HapMap_S28_L001_R1_001.fastq.gz
Found file hugelymodelbat/2019-05-08/HHT5NDSXX_HapMap_S28_L001_R2_001.fastq.gz
Case justhusky exists in Status DB
Destination path already exists: /home/proj/stage/rare-disease/cases/justhusky/wgs/earlycasualcaiman/fastq/1_171015_HHT5NDSXX_earlycasualcaiman_XXXXXX_1.fastq.gz
Destination path already exists: /home/proj/stage/rare-disease/cases/justhusky/wgs/earlycasualcaiman/fastq/1_171015_HHT5NDSXX_earlycasualcaiman_XXXXXX_2.fastq.gz
Destination path already exists: /home/proj/stage/rare-disease/cases/justhusky/wgs/slowlycivilbuck/fastq/1_171015_HHT5NDSXX_slowlycivilbuck_XXXXXX_1.fastq.gz
Destination path already exists: /home/proj/stage/rare-disease/cases/justhusky/wgs/slowlycivilbuck/fastq/1_171015_HHT5NDSXX_slowlycivilbuck_XXXXXX_2.fastq.gz
Destination path already exists: /home/proj/stage/rare-disease/cases/justhusky/wgs/hugelymodelbat/fastq/1_171015_HHT5NDSXX_hugelymodelbat_XXXXXX_1.fastq.gz
Destination path already exists: /home/proj/stage/rare-disease/cases/justhusky/wgs/hugelymodelbat/fastq/1_171015_HHT5NDSXX_hugelymodelbat_XXXXXX_2.fastq.gz
Case justhusky exists in Status DB
Running command /home/proj/stage/bin/miniconda3/envs/S_scout/bin/scout --config /home/proj/stage/servers/config/hasta.scilifelab.se/scout-stage.yaml export panel --bed BRAIN Cardiology CILM CH CTD DIAB ENDO EP HEARING HYDRO IBMFS IEM IF mcarta MHT MIT MOVE mtDNA NBS-M NEURODEG NMD OMIM-AUTO PANELAPP-GREEN PEDHEP PID PIDCAD RETINA SKD SOVM STROKE AID --build 37
Case justhusky exists in Status DB
Running command /home/proj/stage/bin/miniconda3/envs/S_scout/bin/scout --config /home/proj/stage/servers/config/hasta.scilifelab.se/scout-stage.yaml export managed --build 37
Case justhusky exists in Status DB
Config file saved to /home/proj/stage/rare-disease/cases/justhusky/pedigree.yaml
Case justhusky exists in Status DB
earlycasualcaiman: downsampled/external - skip evaluation
Running command /home/proj/stage/bin/miniconda3/bin/conda run --name S_mip12.0 /home/proj/stage/bin/miniconda3/envs/S_mip12.0/bin/mip analyse rd_dna --config /home/proj/stage/servers/config/hasta.scilifelab.se/mip12.0-dna-stage.yaml justhusky --slurm_quality_of_service high --email [email protected] --dry_run_all --qccollect_skip_evaluation

henrikstranneheim · 2023-12-08T14:53:15Z

cg workflow raredisease managed-variants 👍

[henrik.stranneheim@hasta:~] [S_base] 1m47s $ cg workflow raredisease --help
Usage: cg workflow raredisease [OPTIONS] COMMAND [ARGS]...

  NF-core/raredisease analysis workflow.

Options:
  --help  Show this message and exit.

Commands:
  managed-variants  Write managed variants file exported from Scout.
  panel             Write aggregated gene panel file exported from Scout.
θ60° [henrik.stranneheim@hasta:~] [S_base] 24s $ cg workflow raredisease managed-variants --help
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Called undefined __fields__ on HousekeeperAPI, please wrap
Usage: cg workflow raredisease managed-variants [OPTIONS] CASE_ID

  Write managed variants file exported from Scout.

Options:
  -d, --dry-run  Simulate process without executing
  --help         Show this message and exit.

jemten

Nice!

henrikstranneheim · 2023-12-11T07:47:41Z

Deployed in production:

Log deploy... done.
cg, version 54.2.0

feat(WIP):

baa083a

henrikstranneheim added the Project Task Part of a quarterly project label Dec 7, 2023

henrikstranneheim self-assigned this Dec 7, 2023

henrikstranneheim linked an issue Dec 7, 2023 that may be closed by this pull request

Export managed variants from scout #2697

Closed

henrikstranneheim added 3 commits December 8, 2023 09:24

feat(WIP):

4364501

feat(MIP):

4397dee

Merge branch 'master' into add-managed-variant-scout-export

beb14cd

henrikstranneheim marked this pull request as ready for review December 8, 2023 10:16

henrikstranneheim requested a review from a team as a code owner December 8, 2023 10:16

henrikstranneheim added 2 commits December 8, 2023 11:27

feat(refactor):

c1f62ee

Merge remote-tracking branch 'origin/add-managed-variant-scout-export…

fdb95e5

…' into add-managed-variant-scout-export

ChrOertlin approved these changes Dec 8, 2023

View reviewed changes

diitaz93 approved these changes Dec 8, 2023

View reviewed changes

cg/meta/workflow/raredisease.py Show resolved Hide resolved

cg/meta/workflow/raredisease.py Outdated Show resolved Hide resolved

henrikstranneheim added 7 commits December 8, 2023 13:31

feat(refactor):

47e6981

feat(refactor):

abbce90

feat(refactor):

e37bb6d

feat(refactor):

34f1143

feat(refactor):

5d09a53

Merge branch 'master' into add-managed-variant-scout-export

da60e3f

Merge branch 'master' into add-managed-variant-scout-export

1fd7812

henrikstranneheim requested review from jemten and rannick December 8, 2023 14:54

jemten approved these changes Dec 8, 2023

View reviewed changes

henrikstranneheim merged commit 1049038 into master Dec 11, 2023
9 checks passed

henrikstranneheim deleted the add-managed-variant-scout-export branch December 11, 2023 07:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add export managed variants #2750

Add export managed variants #2750

henrikstranneheim commented Dec 7, 2023 •

edited

Loading

ChrOertlin left a comment

diitaz93 left a comment

sonarqubecloud bot commented Dec 8, 2023

henrikstranneheim commented Dec 8, 2023

henrikstranneheim commented Dec 8, 2023

jemten left a comment

henrikstranneheim commented Dec 11, 2023

Add export managed variants #2750

Add export managed variants #2750

Conversation

henrikstranneheim commented Dec 7, 2023 • edited Loading

Description

Added

Changed

Fixed

How to prepare for test

How to test

Expected test outcome

Review

This version is a

Implementation Plan

ChrOertlin left a comment

Choose a reason for hiding this comment

diitaz93 left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Dec 8, 2023

henrikstranneheim commented Dec 8, 2023

cg workflow mip-dna panel justhusky 👍

cg workflow mip-dna managed-variants justhusky 👍

cg workflow mip-dna start justhusky 👍

henrikstranneheim commented Dec 8, 2023

cg workflow raredisease managed-variants 👍

jemten left a comment

Choose a reason for hiding this comment

henrikstranneheim commented Dec 11, 2023

henrikstranneheim commented Dec 7, 2023 •

edited

Loading