Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(Archiving) Clean retrieved springs #2815

Merged
merged 30 commits into from
Jan 12, 2024
Merged

(Archiving) Clean retrieved springs #2815

merged 30 commits into from
Jan 12, 2024

Conversation

islean
Copy link
Contributor

@islean islean commented Jan 9, 2024 •
edited
Loading

Description

When we have retrieved files via Miria, we would want to remove them after some time has passed. This PR adds a CLI for this, which removes Spring files which were retrieved more than 7 days ago and rests their retrieval data, i.e. retrieval_task_id and retrieved_at will be set to None for all files. The data is reset to ensure that the files can be retrieved again.

Added

  • CLI command for deleting retrieved Spring files.

How to prepare for test

  • Ssh to relevant server (depending on type of change)
  • Use stage: us
  • Paxa the environment: paxa
  • Install on stage (example for Hasta): 
    bash /home/proj/production/servers/resources/hasta.scilifelab.se/update-tool-stage.sh -e S_cg -t cg -b archiving-clean-retrieved-springs -a

How to test

  • Do ...

Expected test outcome

  • Check that ...
  • Take a screenshot and attach or copy/paste the output.

Review

  • Tests executed by
  • "Merge and deploy" approved by
    Thanks for filling in who performed the code review and the test!

This version is a

  • MAJOR - when you make incompatible API changes
  • MINOR - when you add functionality in a backwards compatible manner
  • PATCH - when you make backwards compatible bug fixes or documentation/instructions

Implementation Plan

  • Document in ...
  • Deploy this branch on ...
  • Inform to ...

@islean islean self-assigned this Jan 9, 2024
@islean islean requested a review from a team as a code owner January 9, 2024 15:43
@islean islean mentioned this pull request Jan 9, 2024
Closed
22 tasks
@sonarqubecloud SonarQubeCloud
Copy link

sonarqubecloud bot commented Jan 9, 2024

Quality Gate Passed Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

@islean islean marked this pull request as draft January 9, 2024 15:44
@islean islean marked this pull request as ready for review January 10, 2024 11:42
diitaz93
diitaz93 approved these changes Jan 10, 2024
View reviewed changes
Copy link
Contributor

@diitaz93 diitaz93 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Some comments regarding some easy fixable things

cg/apps/housekeeper/hk.py Outdated Show resolved Hide resolved
cg/cli/clean.py Outdated Show resolved Hide resolved
cg/meta/clean/clean_retrieved_spring_files.py Outdated Show resolved Hide resolved
cg/cli/clean.py Outdated Show resolved Hide resolved
cg/meta/clean/clean_retrieved_spring_files.py Outdated Show resolved Hide resolved
tests/meta/clean/conftest.py Outdated Show resolved Hide resolved
tests/meta/clean/conftest.py Outdated Show resolved Hide resolved
tests/meta/clean/conftest.py Show resolved Hide resolved
tests/meta/clean/conftest.py Outdated Show resolved Hide resolved
tests/meta/clean/test_clean_retrieved_spring_files.py Outdated Show resolved Hide resolved
ChrOertlin
ChrOertlin approved these changes Jan 10, 2024
View reviewed changes
Copy link
Contributor

@ChrOertlin ChrOertlin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

cg/apps/housekeeper/hk.py Show resolved Hide resolved
cg/meta/clean/clean_retrieved_spring_files.py Outdated Show resolved Hide resolved
tests/meta/clean/conftest.py Outdated Show resolved Hide resolved
@islean islean requested a review from Vince-janv January 10, 2024 16:02
Vince-janv
Vince-janv approved these changes Jan 11, 2024
View reviewed changes
Copy link
Contributor

@Vince-janv Vince-janv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice ⭐

cg/meta/clean/clean_retrieved_spring_files.py Outdated Show resolved Hide resolved
cg/apps/housekeeper/hk.py Outdated Show resolved Hide resolved
cg/meta/clean/clean_retrieved_spring_files.py Outdated Show resolved Hide resolved
@islean
Copy link
Contributor Author

islean commented Jan 11, 2024 •
edited
Loading

Test performed on stage:

Given an archive entry corresponding to a file which is present on Hasta, and which has a retrieved_at older than a week:

image image

When running cg clean retrieved-spring-files:

image

File is removed:

image

And its retrieval data is reset:

image

@islean
Copy link
Contributor Author

islean commented Jan 11, 2024

Test with file which is marked as retrieved only 2 days ago, meaning that it should not be removed:

File still exists:

1d [isak.ohlsson@hasta:/home/proj/stage/housekeeper-bundles/isaks_test_sample/2024-01-09] [S_base] $ touch isaks_test_spring_file.spring
θ60° 1d [isak.ohlsson@hasta:/home/proj/stage/housekeeper-bundles/isaks_test_sample/2024-01-09] [S_base] $ cg --log-level DEBUG clean retrieved-spring-files
Instantiating housekeeper api
Initializing Store
Starting cleaning of retrieved Spring files.
1d [isak.ohlsson@hasta:/home/proj/stage/housekeeper-bundles/isaks_test_sample/2024-01-09] [S_base] 24s $ ls
isaks_test_spring_file.fastq.gz  isaks_test_spring_file_R1_001.fastq.gz  isaks_test_spring_file_R2_001.fastq.gz  isaks_test_spring_file.spring
1d [isak.ohlsson@hasta:/home/proj/stage/housekeeper-bundles/isaks_test_sample/2024-01-09] [S_base] $

Database entry intact:

image

@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Passed Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

@islean islean merged commit 138cee2 into master Jan 12, 2024
9 checks passed
@islean islean deleted the archiving-clean-retrieved-springs branch January 12, 2024 09:03
@islean
Copy link
Contributor Author

islean commented Jan 12, 2024

Deployed to production:

Getting deployer... done.
Getting last commit message and SHA... done.
Getting version of deploy scripts... /home/isak.ohlsson
done.
Log deploy... done.
cg, version 55.3.0
θ64° 1d [isak.ohlsson@hasta:~] [P_base] 2m44s $

@islean islean mentioned this pull request Jan 19, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@diitaz93 diitaz93 diitaz93 approved these changes

@Vince-janv Vince-janv Vince-janv approved these changes

@ChrOertlin ChrOertlin ChrOertlin approved these changes

Assignees

@islean islean

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@islean @ChrOertlin @diitaz93 @Vince-janv