(Archiving) Use tag when fetching spring files

[islean]

Description

Closes #2729. The intent is to speed up the archive CLI commands by filtering the spring files we fetch with the newly added archive location tag.

Changed

• Files to archive are now filtered already in the SQL query, using the archive location tag.

How to prepare for test

• Ssh to relevant server (depending on type of change)
• Use stage: us
• Paxa the environment: paxa
• Install on stage (example for Hasta):

  bash /home/proj/production/servers/resources/hasta.scilifelab.se/update-tool-stage.sh -e S_cg -t cg -b archiving-use-location-tag -a

How to test

• Do ...

Expected test outcome

• Check that ...
• Take a screenshot and attach or copy/paste the output.

Review

• Tests executed by
• "Merge and deploy" approved by
  Thanks for filling in who performed the code review and the test!

This version is a

• MAJOR - when you make incompatible API changes
• MINOR - when you add functionality in a backwards compatible manner
• PATCH - when you make backwards compatible bug fixes or documentation/instructions

Implementation Plan

• Document in ...
• Deploy this branch on ...
• Inform to ...

[islean]

Archiving three files using the tag reduced the run time from 20 minutes to 30 seconds:

θ63° 1d [isak.ohlsson@hasta:~] [S_base] 3m50s $ cg archive archive-spring-files
/home/proj/stage/bin/miniconda3/envs/S_cg/lib/python3.11/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host 'cg-df-cat.scilifelab.se'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Sending request with headers: 
{'Content-Type': 'application/json', 'accept': 'application/json', 'Authorization': 'Bearer ****'} 
and body: 
{'osType': 'Unix/MacOS', 'createFolder': True, 'settings': [], 'pathInfo': [{'destination': 'archive@SciLifeLab:/stage/isaks_test_bundle', 'source': 'gpfs@Hasta:/proj/stage/housekeeper-bundles/isaks_test_bundle/2023-11-30/isaks_test_spring_file.spring'}], 'metadataList': []}
/home/proj/stage/bin/miniconda3/envs/S_cg/lib/python3.11/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host 'cg-df-cat.scilifelab.se'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Files submitted to karolinska_bucket with archival task id 132.
Adding an archive to file 5002316 with archiving task id 132
θ62° 1d [isak.ohlsson@hasta:~] [S_base] 28s $ 

[islean]

Updating job statuses ran for 27 seconds as well:

θ60° 1d [isak.ohlsson@hasta:~] [S_base] 24s 2 $ cg archive update-job-statuses
/home/proj/stage/bin/miniconda3/envs/S_cg/lib/python3.11/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host 'cg-df-cat.scilifelab.se'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Fetching status for job with id 132 from karolinska_bucket
Sending request with headers: 
{'Content-Type': 'application/json', 'accept': 'application/json', 'Authorization': 'Bearer ****'} 
and body: 
{'id': 132}
/home/proj/stage/bin/miniconda3/envs/S_cg/lib/python3.11/site-packages/urllib3/connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host 'cg-df-cat.scilifelab.se'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Miria returned status Completed for job 132
Job with id 132 has finished, updating Archive entries.
θ60° 1d [isak.ohlsson@hasta:~] [S_base] 27s $ 

[ChrOertlin]

💯

[Vince-janv]

Nice solution 👍

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[islean]

Deployed to production:

Log deploy... done.
cg, version 53.4.3