DEV - Update pacbio flow to accept barcoded samples #3754

diitaz93 · 2024-09-23T10:56:53Z

Description

Added

Changed

Pacbio runs are expected to be barcoded.

Fixed

How to prepare for test

Ssh to relevant server (depending on type of change)
Use stage: us
Paxa the environment: paxa

Install on stage (example for Hasta):

bash /home/proj/production/servers/resources/hasta.scilifelab.se/update-tool-stage.sh -e S_cg -t cg -b dev-update-pacbio-models -a

How to test

See https://github.com/Clinical-Genomics/add-new-tech/issues/116

Review

Tests executed by IO, SD
"Merge and deploy" approved by VJ
Thanks for filling in who performed the code review and the test!

This version is a

MINOR - when you add functionality in a backwards compatible manner

Implementation Plan

Document in ...
Deploy this branch on ...
Inform to ...

cg/store/models.py

### Changed - Barcode fields added to the PacBioSequencingRun model

* add reports and reads for barcoded run * reformat json * add some fixtures and constant * renamed fixture names * fix failing test

cg/store/models.py

### Added - BarcodeMetrics class

### Added - SampleMetrics parsing model

* Include barcodes report in the file manager * Move fixtures to plugins * parametrise tests to include new run * Update cg/services/run_devices/pacbio/run_file_manager/run_file_manager.py

* Include barcodes report in the file manager * Move fixtures to plugins * parametrise tests to include new run * add functionality to parse barcode metrics * add support for parsing barcodes metrics * address comments by IO * remove from_attributes

cg/constants/pacbio.py

* remove sample_id fetching from dataset metrics * fixed metrics parsing tests and fixtures * removed unused fixtures * move fixtures from conftest to plugins * add test for new function

cg/services/run_devices/pacbio/data_transfer_service/utils.py

cg/services/run_devices/pacbio/metrics_parser/models.py

* commit to initialise branch * update DTOs and CRUD * move dto fixtures to plugins * update fixtures and fix tests * remove duplicated fixture and unused inits * fix import error

* Add upgrade migration * add downgrade * fix black * add None to new fields on the db --------- Co-authored-by: Sebastian Diaz <[email protected]>

islean · 2024-09-30T13:59:36Z

Upgrade and deployment to stage run successfully:

remote: Compressing objects: 100% (42/42), done.
remote: Total 94 (delta 56), reused 83 (delta 52), pack-reused 0 (from 0)
Unpacking objects: 100% (94/94), done.
From github.com:Clinical-Genomics/cg
3c20481..0912472 3767-allow-micro-seq-only-excel -> origin/3767-allow-micro-seq-only-excel
da6ab10..6d6dd60 add-raredisease-scout-upload -> origin/add-raredisease-scout-upload
722338b..fcce1fa add-tomte-delivery-report-to-scout -> origin/add-tomte-delivery-report-to-scout

[new branch] allow-mip-rna-json -> origin/allow-mip-rna-json
3897644..691055f clean-case-dirs -> origin/clean-case-dirs
0c0c38a..78dc316 dev-update-pacbio-models -> origin/dev-update-pacbio-models
8093891..8693073 fix-raredisease-sample-pairs-list -> origin/fix-raredisease-sample-pairs-list
Branch dev-update-pacbio-models set up to track remote branch dev-update-pacbio-models from origin.
Switched to a new branch 'dev-update-pacbio-models'
Already up-to-date.
/home/isak.ohlsson
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade 18dbadd8c436 -> d6474325c722, Refactor pacbio models
fatal: Not a git repository (or any parent up to mount point /home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
repository is clean
Logging deploy ...
Getting deployer... done.
Getting last commit message and SHA... done.
Getting version of deploy scripts... /home/isak.ohlsson
done.
Log deploy... done.
cg, version 63.2.25
[isak.ohlsson@hasta:~] [S_base] $

islean · 2024-09-30T14:00:32Z

Barcoded columns added but potentially wrong database type? Do we need bigint?

islean · 2024-09-30T14:01:00Z

islean · 2024-09-30T14:17:54Z

Barcoded columns added but potentially wrong database type? Do we need bigint?

Yes, we need it. Post processing failed otherwise

islean · 2024-09-30T14:22:44Z

Downgrade successful:

Readded columns have wrong type (int instead of bigint). Revision has been changed to address this

islean · 2024-09-30T14:28:03Z

Correct types added:

Vince-janv

Can't believe how quick this went 🤩 Well done 💪

alembic/versions/2024_09_27_d6474325c722_refactor_pacbio_models.py

cg/store/models.py

cg/constants/pacbio.py

cg/store/models.py

* Removed fixtures from unbarcoded cell * Remove smrt cell files * Improve test description

* Removed fixtures from unbarcoded cell * Remove smrt cell files * Improve test description * add run name to database

* remove barcode from database * update alembic script

### Fixed - Fixed typo in alias

* create update_sample_reads function in CRUD and add unit test * fix relative import * parametrise update_last_sequenced_at unit test * add sample updating to post-processing and update unit test * minor name fixes * address comments by IO

### Changed - sample_name to sample_internal_id - Unassigned reads are disregarded.

sonarqubecloud · 2024-10-02T14:58:42Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

islean · 2024-10-03T07:47:40Z

Deployed to production:

 create mode 100644 tests/fixtures/devices/pacbio/SMRTcells/r84202_20240913_121403/1_C01/statistics/unzipped_reports/barcodes.report.json
 rename tests/fixtures/devices/pacbio/SMRTcells/{r84202_20240522_133539/1_A01 => r84202_20240913_121403/1_C01}/statistics/unzipped_reports/control.report.json (87%)
 rename tests/fixtures/devices/pacbio/SMRTcells/{r84202_20240522_133539/1_A01 => r84202_20240913_121403/1_C01}/statistics/unzipped_reports/loading.report.json (87%)
 rename tests/fixtures/devices/pacbio/SMRTcells/{r84202_20240522_133539/1_A01 => r84202_20240913_121403/1_C01}/statistics/unzipped_reports/raw_data.report.json (88%)
 create mode 100644 tests/fixtures/devices/pacbio/SMRTcells/r84202_20240913_121403/1_C01/statistics/unzipped_reports/smrtlink-datasets.json
 delete mode 100644 tests/services/run_devices/__init__.py
 delete mode 100644 tests/services/run_devices/pacbio/__init__.py
 delete mode 100644 tests/services/run_devices/pacbio/housekeeper_service/conftest.py
 delete mode 100644 tests/services/run_devices/pacbio/metrics_parser/__init__.py
/home/isak.ohlsson
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 18dbadd8c436 -> d6474325c722, Refactor pacbio models
[isak.ohlsson@hasta:~] [P_base] $

diitaz93 added 2 commits September 23, 2024 12:55

commit to initialise branch, add todos

048fc67

commit to initialise branch, add todos

536a550

diitaz93 changed the title ~~DEV - Update pacbio models to accepts barcoded samples~~ DEV - Update pacbio flow to accept barcoded samples Sep 23, 2024

diitaz93 assigned diitaz93 and islean Sep 23, 2024

diitaz93 added alembic migration Perform a change in database feature New feature labels Sep 23, 2024

Update db model PacBioSampleSequencingMetrics (#3756)

9925c2f

diitaz93 commented Sep 23, 2024

View reviewed changes

cg/store/models.py Outdated Show resolved Hide resolved

islean and others added 2 commits September 23, 2024 14:04

DEV Update PacbioSequencingRun model (#3755) (patch)

2457d18

### Changed - Barcode fields added to the PacBioSequencingRun model

Add fixtures for barcoded runs (#3757)

f1ab214

* add reports and reads for barcoded run * reformat json * add some fixtures and constant * renamed fixture names * fix failing test

diitaz93 commented Sep 23, 2024

View reviewed changes

cg/store/models.py Outdated Show resolved Hide resolved

islean and others added 6 commits September 23, 2024 15:25

DEV Add model for parsing barcode data (#3758) (patch)

c51c317

### Added - BarcodeMetrics class

DEV Add sample metrics parsing model (#3759) (patch)

1b48881

### Added - SampleMetrics parsing model

Add barcodes report to file manager (#3760)

ddba217

* Include barcodes report in the file manager * Move fixtures to plugins * parametrise tests to include new run * Update cg/services/run_devices/pacbio/run_file_manager/run_file_manager.py

Merge branch 'master' into dev-update-pacbio-models

80f0d2a

Merge branch 'master' into dev-update-pacbio-models

ddbce41

diitaz93 commented Sep 26, 2024

View reviewed changes

cg/constants/pacbio.py Outdated Show resolved Hide resolved

Fix dataset metrics parsing (#3773)

5648853

* remove sample_id fetching from dataset metrics * fixed metrics parsing tests and fixtures * removed unused fixtures * move fixtures from conftest to plugins * add test for new function

diitaz93 commented Sep 27, 2024

View reviewed changes

cg/services/run_devices/pacbio/data_transfer_service/utils.py Outdated Show resolved Hide resolved

Merge branch 'master' into dev-update-pacbio-models

495d098

diitaz93 commented Sep 27, 2024

View reviewed changes

cg/services/run_devices/pacbio/metrics_parser/models.py Show resolved Hide resolved

diitaz93 and others added 5 commits September 27, 2024 15:37

Update Pacbio DTOs with barcode info (#3778)

6865ea9

* commit to initialise branch * update DTOs and CRUD * move dto fixtures to plugins * update fixtures and fix tests * remove duplicated fixture and unused inits * fix import error

Merge branch 'master' into dev-update-pacbio-models

982ed7b

DEV Pacbio alembic script (#3786)

e3790ef

* Add upgrade migration * add downgrade * fix black * add None to new fields on the db --------- Co-authored-by: Sebastian Diaz <[email protected]>

add logging to decompression

0c0c38a

Merge branch 'master' into dev-update-pacbio-models

78dc316

Integer to bigint

0db6ffc

Vince-janv approved these changes Sep 30, 2024

View reviewed changes

alembic/versions/2024_09_27_d6474325c722_refactor_pacbio_models.py Outdated Show resolved Hide resolved

cg/store/models.py Outdated Show resolved Hide resolved

cg/constants/pacbio.py Outdated Show resolved Hide resolved

cg/store/models.py Show resolved Hide resolved

islean and others added 10 commits October 1, 2024 09:58

Merge branch 'master' into dev-update-pacbio-models

ebe3c2b

Clean pacbio fixtures (#3794)

5b213d5

* Removed fixtures from unbarcoded cell * Remove smrt cell files * Improve test description

Add run_name to pacbio sequencing run model (#3795)

52f90aa

* Removed fixtures from unbarcoded cell * Remove smrt cell files * Improve test description * add run name to database

Merge branch 'master' into dev-update-pacbio-models

500f621

Remove barcodes db (#3800)

43935df

* remove barcode from database * update alembic script

Fix typo in alias (#3801) (patch)

dbfaeea

### Fixed - Fixed typo in alias

typed arg in alembic script

a8840c4

Add handling of unassigned reads (#3803) (patch)

addcb6f

### Changed - sample_name to sample_internal_id - Unassigned reads are disregarded.

Standarise constants for aliases

f589259

diitaz93 marked this pull request as ready for review October 2, 2024 14:57

diitaz93 requested a review from a team as a code owner October 2, 2024 14:57

islean merged commit fa45dce into master Oct 3, 2024
9 checks passed

islean deleted the dev-update-pacbio-models branch October 3, 2024 07:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DEV - Update pacbio flow to accept barcoded samples #3754

DEV - Update pacbio flow to accept barcoded samples #3754

diitaz93 commented Sep 23, 2024 •

edited by islean

Loading

islean commented Sep 30, 2024

islean commented Sep 30, 2024

islean commented Sep 30, 2024

islean commented Sep 30, 2024

islean commented Sep 30, 2024

islean commented Sep 30, 2024

Vince-janv left a comment

sonarqubecloud bot commented Oct 2, 2024

islean commented Oct 3, 2024

DEV - Update pacbio flow to accept barcoded samples #3754

DEV - Update pacbio flow to accept barcoded samples #3754

Conversation

diitaz93 commented Sep 23, 2024 • edited by islean Loading

Description

Added

Changed

Fixed

How to prepare for test

How to test

Review

This version is a

Implementation Plan

islean commented Sep 30, 2024

islean commented Sep 30, 2024

islean commented Sep 30, 2024

islean commented Sep 30, 2024

islean commented Sep 30, 2024

islean commented Sep 30, 2024

Vince-janv left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Oct 2, 2024

Quality Gate passed

islean commented Oct 3, 2024

diitaz93 commented Sep 23, 2024 •

edited by islean

Loading