Add automatic quality control of analyses for microbial samples #2754

seallard · 2023-12-08T14:37:40Z

Description

This PR implements automatic QC for microsalt following the documented requirements in Atlas. Part of #1648.

This logic will likely be trashed once the JASEN pipeline replaces microsalt 😅

Originally implemented in #1655 and then disabled in #2505.

Description of implementation

The flow is the following:

Given a list of cases which are ready for quality control
For each case:
- Check whether the quality control is necessary for the case
For each sample in the case
- Check whether the sample pass the criteria listed in the documentation.
- The failed samples are tracked for the overall case evaluation.
Check whether the case pass the criteria listed in the documentation.
If the case fails, set status to failed in trailblazer.
If the case pass, include it in the cases to store.

Description of implemented sample quality controls

The reads passes > 70% of the application target reads
The reads for the negative control < 20 % of the application target reads
The mapping rate passes > 30 %
The duplication rate is < 80 %
The median insert size is > 100 bp
The average coverage is > 10x
The % BP with 10x coverage is > 75 %

Description of implemented case quality controls

The negative control sample pass the quality control
All urgent samples ("MWR") pass the quality control
At least 90 % of the non urgent samples ("MWX") pass the quality control

Questions

Do we want to change the threshold for the reads? Yes, skip constant and use percentage guaranteed reads.
Will all microsalt cases have a negative control? No
Can a case contain a mix of urgent and non urgent samples? If so, what validation rules should we apply? No.
What does expected reads mean and how does it differ from target reads for an application? Use target reads.
Do you want to change something about the structure of the generated report? Add summary to top of report.
Any additional information in trailblazer? Add short summary regardless of pass/fail.

Tested in stage on Hasta with cg workflow microsalt qc-microsalt gamelizard

These were implemented in #1655 and reverted in #2505.

seallard · 2023-12-18T13:00:00Z

Blocked until meeting with production and help with manual testing.

seallard · 2023-12-19T10:05:22Z

Tested in stage

[sebastian.allard@hasta:/home/proj/stage/microbial/results] [S_base] 1 $ cg workflow microsalt qc-microsalt gamelizard
Performing QC on case gamelizard
ACC10005A1 passed QC.
Control sample ACC10005A2 passed QC.
ACC10005A3 passed QC.
ACC10005A4 passed QC.
ACC10005A5 passed QC.
ACC10005A6 passed QC.
ACC10005A7 passed QC.
ACC10005A8 passed QC.
ACC10005A9 passed QC.
ACC10005A10 passed QC.
ACC10005A11 passed QC.
ACC10005A12 passed QC.
ACC10005A13 passed QC.
ACC10005A14 passed QC.
ACC10005A15 passed QC.
ACC10005A16 passed QC.
ACC10005A17 passed QC.
ACC10005A18 passed QC.
ACC10005A19 passed QC.
QC passed, see /home/proj/stage/microbial/results/ACC10005A14_2022.7.26_13.1.1/QC_done.json for details.
Sample results: 0 failed, 19 passed, 19 total.

The generated qc report:

{
    "case": {
        "passes_qc": true,
        "control_passes_qc": true,
        "urgent_passes_qc": true,
        "non_urgent_passes_qc": true
    },
    "samples": [{
        "sample_id": "ACC10005A1",
        "passes_qc": true,
        "is_control": false,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }, {
        "sample_id": "ACC10005A2",
        "passes_qc": true,
        "is_control": true,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }, {
        "sample_id": "ACC10005A3",
        "passes_qc": true,
        "is_control": false,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }, {
        "sample_id": "ACC10005A4",
        "passes_qc": true,
        "is_control": false,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }, {
        "sample_id": "ACC10005A5",
        "passes_qc": true,
        "is_control": false,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }, {
        "sample_id": "ACC10005A6",
        "passes_qc": true,
        "is_control": false,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }, {
        "sample_id": "ACC10005A7",
        "passes_qc": true,
        "is_control": false,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }, {
        "sample_id": "ACC10005A8",
        "passes_qc": true,
        "is_control": false,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }, {
        "sample_id": "ACC10005A9",
        "passes_qc": true,
        "is_control": false,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }, {
        "sample_id": "ACC10005A10",
        "passes_qc": true,
        "is_control": false,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }, {
        "sample_id": "ACC10005A11",
        "passes_qc": true,
        "is_control": false,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }, {
        "sample_id": "ACC10005A12",
        "passes_qc": true,
        "is_control": false,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }, {
        "sample_id": "ACC10005A13",
        "passes_qc": true,
        "is_control": false,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }, {
        "sample_id": "ACC10005A14",
        "passes_qc": true,
        "is_control": false,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }, {
        "sample_id": "ACC10005A15",
        "passes_qc": true,
        "is_control": false,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }, {
        "sample_id": "ACC10005A16",
        "passes_qc": true,
        "is_control": false,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }, {
        "sample_id": "ACC10005A17",
        "passes_qc": true,
        "is_control": false,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }, {
        "sample_id": "ACC10005A18",
        "passes_qc": true,
        "is_control": false,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }, {
        "sample_id": "ACC10005A19",
        "passes_qc": true,
        "is_control": false,
        "application_tag": "MWRNXTR003",
        "passes_reads_qc": true,
        "passes_mapping_qc": true,
        "passes_duplication_qc": true,
        "passes_inserts_qc": true,
        "passes_coverage_qc": true,
        "passes_10x_coverage_qc": true
    }]
}

karlnyr

Very nice work, I have some changes I would like to see before we implement this into production.

cg/meta/workflow/microsalt/quality_controller/quality_controller.py

cg/constants/constants.py

cg/meta/workflow/microsalt/quality_controller/utils.py

cg/meta/workflow/microsalt/constants.py

cg/meta/workflow/microsalt/metrics_parser/models.py

cg/meta/workflow/microsalt/microsalt.py

tests/meta/workflow/test_microsalt.py

seallard · 2024-01-02T09:27:31Z

Tested in stage

[sebastian.allard@hasta:~] [S_base] 22s 2 $ cg workflow microsalt qc gamelizard
Performing QC on case gamelizard
ACC10005A1 passed QC.
Control sample ACC10005A2 passed QC.
ACC10005A3 passed QC.
ACC10005A4 passed QC.
ACC10005A5 passed QC.
ACC10005A6 passed QC.
ACC10005A7 passed QC.
ACC10005A8 passed QC.
ACC10005A9 passed QC.
ACC10005A10 passed QC.
ACC10005A11 passed QC.
ACC10005A12 passed QC.
ACC10005A13 passed QC.
ACC10005A14 passed QC.
ACC10005A15 passed QC.
ACC10005A16 passed QC.
ACC10005A17 passed QC.
ACC10005A18 passed QC.
ACC10005A19 passed QC.
QC passed, see /home/proj/stage/microbial/results/ACC10005A14_2022.7.26_13.1.1/QC_done.json for details.
Sample results: 0 failed, 19 passed, 19 total

sonarqubecloud · 2024-01-02T09:28:38Z

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

seallard · 2024-01-02T09:49:16Z

Deployed to prod Hasta

…(minor)

seallard and others added 30 commits December 7, 2023 16:51

Add microsalt qc checks

3abe921

These were implemented in #1655 and reverted in #2505.

Extract qc logic to separate module

c3f669e

Merge branch 'master' into add-microsalt-qc

7da4955

Fix imports

0d79d1e

Merge branch 'master' into add-microsalt-qc

139c888

Formatting

1a4d661

Remove failing test

da79782

Add sample total reads check

2f34f6d

Fix tests

79feb5c

Quality control for total reads given sample id

09ddae9

Validate negative control total reads

cca1d73

Add models for parsing quality metrics

09830e6

Validate mapped rate

0c3aa9b

Add validation of duplication rate

418837d

Validate median insert size

4cabeca

Validate average coverage

72b3f99

Validate 10x coverage

71634c0

Formatting

0227515

Validate that negative control sample for cases passes control

48e247b

Fix quality result model

ec60b96

Validate that all urgent samples pass quality control

ea264ca

Validate that most non urgent samples pass qc

5da77a4

Handle div by zero

d33a84c

Remove deprecated code

738fa8b

Fix parsing of metrics

dbd596f

Pass metrics file path as single parameter to qc

c118eef

Handle possible null values from metrics

296b12f

Restructure methods

603c3a5

Restructure microsalt module

b1750d0

Generate quality report

151b157

seallard added 3 commits December 18, 2023 13:28

Add logging

f52c801

Improve messages

f485804

Improve logging

a6df7b8

seallard added 4 commits December 18, 2023 16:12

Simplify run dir retrieval

f946950

Fix test

5ad401a

Add log messages

d62abd9

Add logging

6f0a0ab

seallard added 2 commits December 19, 2023 11:24

Improve logging

f152d2d

Fix exc

bc1b6cb

Clinical-Genomics deleted a comment from sonarqubecloud bot Dec 19, 2023

karlnyr requested changes Dec 20, 2023

View reviewed changes

seallard added 8 commits December 22, 2023 16:23

Fix comments

fde8536

Handle missing metrics file

9900e26

Handle missing negative sample

517fe95

Use application guaranteed reads

8ab1cbe

Fix tests

0538866

Fix name for microsalt qc command

053283f

Add summary to report

4f840d1

Add summary to trailblazer comment

d463810

Merge branch 'master' into add-microsalt-qc

6a4fbda

seallard merged commit 0ae1afe into master Jan 2, 2024
9 checks passed

seallard deleted the add-microsalt-qc branch January 2, 2024 09:38

seallard mentioned this pull request Jan 2, 2024

Add a "validate" step before storage to microSALT and Mutant #1648

Closed

2 tasks

karlnyr pushed a commit that referenced this pull request Jan 31, 2024

Add automatic quality control of analyses for microbial samples (#2754)…

451909f

…(minor)

seallard mentioned this pull request Mar 14, 2024

Empty fastq files in delivery for microsalt cases #2983

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add automatic quality control of analyses for microbial samples #2754

Add automatic quality control of analyses for microbial samples #2754

seallard commented Dec 8, 2023 •

edited

Loading

seallard commented Dec 18, 2023

seallard commented Dec 19, 2023 •

edited

Loading

karlnyr left a comment

seallard commented Jan 2, 2024

sonarqubecloud bot commented Jan 2, 2024

seallard commented Jan 2, 2024

Add automatic quality control of analyses for microbial samples #2754

Add automatic quality control of analyses for microbial samples #2754

Conversation

seallard commented Dec 8, 2023 • edited Loading

Description

Description of implementation

Description of implemented sample quality controls

Description of implemented case quality controls

Questions

seallard commented Dec 18, 2023

seallard commented Dec 19, 2023 • edited Loading

karlnyr left a comment

Choose a reason for hiding this comment

seallard commented Jan 2, 2024

sonarqubecloud bot commented Jan 2, 2024

Quality Gate passed

seallard commented Jan 2, 2024

seallard commented Dec 8, 2023 •

edited

Loading

seallard commented Dec 19, 2023 •

edited

Loading