Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fetching flow cells from PDC fails #2114

Closed
seallard opened this issue Jun 8, 2023 · 8 comments
Closed

Fetching flow cells from PDC fails #2114

seallard opened this issue Jun 8, 2023 · 8 comments

Comments

@seallard
Copy link
Contributor

seallard commented Jun 8, 2023 •
edited
Loading

Description

For two flow cells, it seems like they can not be retrieved from the PDC. Specifically flow cell HHH5TDSX2 and HKJL7DSXY. Here is an example of a log (/home/proj/production/logs/cron-cg-fetch-flowcell.log) where the retrieval of the HHH5TDSX2 flow cell from PDC fails:

2023-03-13 16:43:40 hasta.scilifelab.se cg.meta.backup.backup[171712] INFO HHH5TDSX2: retrieving from PDC
2023-03-13 16:43:40 hasta.scilifelab.se cg.utils.commands[171712] INFO Running command /bin/dsmc q archive /home/hiseq.clinical/ENCRYPT/*HHH5TDSX2*
2023-03-13 16:43:56 hasta.scilifelab.se cg.utils.commands[171712] INFO Running command /bin/dsmc q archive /mnt/hds/proj/bioinfo/TO_PDC/*HHH5TDSX2*
2023-03-13 16:44:01 hasta.scilifelab.se cg.utils.commands[171712] CRITICAL Call ['/bin/dsmc', 'q', 'archive', '/mnt/hds/proj/bioinfo/TO_PDC/*HHH5TDSX2*'] exit with a non zero exit code
2023-03-13 16:44:01 hasta.scilifelab.se cg.utils.commands[171712] CRITICAL ANS1092W No files matching search criteria were found
2023-03-13 16:44:01 hasta.scilifelab.se cg.meta.backup.backup[171712] INFO No archived files found for pdc query ['/home/hiseq.clinical/ENCRYPT/*HHH5TDSX2*', '/mnt/hds/proj/bioinfo/TO_PDC/*HHH5TDSX2*'], testing legacy directory
2023-03-13 16:44:01 hasta.scilifelab.se cg.meta.backup.backup[171712] INFO Found archived files for pdc queries ['/home/hiseq.clinical/ENCRYPT/*HHH5TDSX2*', '/mnt/hds/proj/bioinfo/TO_PDC/*HHH5TDSX2*']
2023-03-13 16:44:01 hasta.scilifelab.se cg.meta.backup.backup[171712] INFO Encryption key found: /home/hiseq.clinical/ENCRYPT/211029_A00621_0522_AHHH5TDSX2.key.gpg
2023-03-13 16:44:01 hasta.scilifelab.se cg.meta.backup.backup[171712] INFO Flow cell found: /home/hiseq.clinical/ENCRYPT/211029_A00621_0522_AHHH5TDSX2.tar.gz.gpg
2023-03-13 16:44:01 hasta.scilifelab.se cg.utils.commands[171712] INFO Running command /bin/dsmc retrieve -replace=yes /home/hiseq.clinical/ENCRYPT/211029_A00621_0522_AHHH5TDSX2.key.gpg /home/proj/production/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.key.gpg
2023-03-13 16:45:31 hasta.scilifelab.se cg.utils.commands[171712] CRITICAL Call ['/bin/dsmc', 'retrieve', '-replace=yes', '/home/hiseq.clinical/ENCRYPT/211029_A00621_0522_AHHH5TDSX2.key.gpg', '/home/proj/production/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.key.gpg'] exit with a non zero exit code
2023-03-13 16:45:31 hasta.scilifelab.se cg.utils.commands[171712] CRITICAL 
2023-03-13 16:45:31 hasta.scilifelab.se cg.meta.backup.backup[171712] WARNING WARNING for retrieval of encryption key of flow cell HHH5TDSX2, please check dsmerror.log
2023-03-13 16:45:31 hasta.scilifelab.se cg.utils.commands[171712] INFO Running command /bin/dsmc retrieve -replace=yes /home/hiseq.clinical/ENCRYPT/211029_A00621_0522_AHHH5TDSX2.tar.gz.gpg /home/proj/production/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.tar.gz.gpg
2023-03-13 18:34:15 hasta.scilifelab.se cg.utils.commands[171712] CRITICAL Call ['/bin/dsmc', 'retrieve', '-replace=yes', '/home/hiseq.clinical/ENCRYPT/211029_A00621_0522_AHHH5TDSX2.tar.gz.gpg', '/home/proj/production/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.tar.gz.gpg'] exit with a non zero exit code
2023-03-13 18:34:15 hasta.scilifelab.se cg.utils.commands[171712] CRITICAL 
2023-03-13 18:34:15 hasta.scilifelab.se cg.meta.backup.backup[171712] WARNING WARNING for retrieval of flow cell HHH5TDSX2, please check dsmerror.log
2023-03-13 18:34:15 hasta.scilifelab.se cg.meta.backup.backup[171712] INFO Status for flow cell HHH5TDSX2 set to retrieved
2023-03-13 18:34:15 hasta.scilifelab.se cg.meta.encryption.encryption[171712] INFO Starting GPG command:
2023-03-13 18:34:15 hasta.scilifelab.se cg.meta.encryption.encryption[171712] INFO /bin/gpg --decrypt --batch --cipher-algo AES256 --passphrase  Secret -o /home/proj/production/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.key /home/proj/production/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.key.gpg
2023-03-13 18:34:15 hasta.scilifelab.se cg.utils.commands[171712] INFO Running command /bin/gpg --decrypt --batch --cipher-algo AES256 --passphrase Secret -o /home/proj/production/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.key /home/proj/production/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.key.gpg
2023-03-13 18:34:15 hasta.scilifelab.se cg.meta.encryption.encryption[171712] INFO Starting GPG command:
2023-03-13 18:34:15 hasta.scilifelab.se cg.meta.encryption.encryption[171712] INFO /bin/gpg --decrypt --cipher-algo AES256 --batch --passphrase-file /home/proj/production/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.key -o /home/proj/production/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.tar.gz /home/proj/productio
n/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.tar.gz.gpg
2023-03-13 18:34:15 hasta.scilifelab.se cg.utils.commands[171712] INFO Running command /bin/gpg --decrypt --cipher-algo AES256 --batch --passphrase-file /home/proj/production/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.key -o /home/proj/production/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.tar.gz /home/proj/produ
ction/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.tar.gz.gpg
2023-03-14 02:49:22 hasta.scilifelab.se cg.utils.commands[171712] CRITICAL Call ['/bin/gpg', '--decrypt', '--cipher-algo', <CipherAlgorithm.AES256: 'AES256'>, '--batch', '--passphrase-file', '/home/proj/production/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.key', '-o', '/home/proj/production/flowcells/novaseq/runs/211029_A
00621_0522_AHHH5TDSX2.tar.gz', '/home/proj/production/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.tar.gz.gpg'] exit with a non zero exit code
2023-03-14 02:49:22 hasta.scilifelab.se cg.utils.commands[171712] CRITICAL gpg: AES256 encrypted data
gpg: encrypted with 1 passphrase
gpg: decryption failed: Bad signature
2023-03-14 02:49:22 hasta.scilifelab.se cg.meta.backup.backup[171712] ERROR Decryption failed: None
Traceback (most recent call last):
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/cg/meta/backup/backup.py", line 122, in _process_flow_cell
    ) = self.decrypt_flow_cell(archived_flow_cell, archived_key, run_dir)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/cg/meta/backup/backup.py", line 201, in decrypt_flow_cell
    self.encryption_api.run_gpg_command(decryption_command)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/cg/meta/encryption/encryption.py", line 34, in run_gpg_command
    self.process.run_command(command, dry_run=self.dry_run)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/cg/utils/commands.py", line 108, in run_command
    raise CalledProcessError(res.returncode, command)
subprocess.CalledProcessError: Command '['/bin/gpg', '--decrypt', '--cipher-algo', <CipherAlgorithm.AES256: 'AES256'>, '--batch', '--passphrase-file', '/home/proj/production/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.key', '-o', '/home/proj/production/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.tar.gz', '/home/pr
oj/production/flowcells/novaseq/runs/211029_A00621_0522_AHHH5TDSX2.tar.gz.gpg']' returned non-zero exit status 2.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/pymysql/connections.py", line 692, in _read_packet
    packet_header = self._read_bytes(4)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/pymysql/connections.py", line 749, in _read_bytes
    CR.CR_SERVER_LOST, "Lost connection to MySQL server during query"
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')

The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/proj/production/bin/miniconda3/envs/P_main/bin/cg", line 8, in <module>
    sys.exit(base())
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/click/decorators.py", line 27, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/cg/cli/backup.py", line 62, in fetch_flow_cell
    retrieval_time: Optional[float] = backup_api.fetch_flow_cell(flow_cell=flow_cell)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/cg/meta/backup/backup.py", line 102, in fetch_flow_cell
    archived_flow_cell=archived_flow_cell,
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/cg/meta/backup/backup.py", line 133, in _process_flow_cell
    self.status.commit()
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/orm/scoping.py", line 163, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1046, in commit
    self.transaction.commit()
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 504, in commit
    self._prepare_impl()
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
    self.session.flush()
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2540, in flush
    self._flush(objects)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2682, in _flush
    transaction.rollback(_capture_exception=True)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    with_traceback=exc_tb,
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2642, in _flush
    flush_context.execute()
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
    rec.execute(self)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute
    uow,
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 236, in save_obj
    update,
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 995, in _emit_update_statements
    statement, multiparams
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
    distilled_params,
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
    e, statement, parameters, cursor, context
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/pymysql/connections.py", line 692, in _read_packet
    packet_header = self._read_bytes(4)
  File "/home/proj/production/bin/miniconda3/envs/P_main/lib/python3.7/site-packages/pymysql/connections.py", line 749, in _read_bytes
    CR.CR_SERVER_LOST, "Lost connection to MySQL server during query"
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
[SQL: UPDATE flowcell SET `status`=%(status)s, updated_at=%(updated_at)s WHERE flowcell.id = %(flowcell_id)s]
[parameters: {'status': 'requested', 'updated_at': datetime.datetime(2023, 3, 14, 2, 49, 22, 585364), 'flowcell_id': 1617}]
(Background on this error at: http://sqlalche.me/e/13/e3q8)

RESULT CODE: 1

Suggested solution

Write here if there is a suggested way to solve the issue

This can be closed when

The flow cells mentioned above can be fetched without any error.

Blocked by

If there are any blocking issues/prs/things in this or other repos. Please link to them.
@seallard
Copy link
Contributor Author

seallard commented Jun 9, 2023 •
edited
Loading

The issue is not reproducible in the stage environment on Hasta

us
cg backup fetch-flow-cell -f HHH5TDSX2

If there is a message like

Processing queue is full
HHH5TDSX2: updating flow cell status to requested

Just find the flow cell in status-db-stage with the processing status and toggle it to retrieved instead and then rerun the backup command.

But a different error is given in the stage environment, not sure what is going on:

HHH5TDSX2: retrieving from PDC
Running command /bin/dsmc q archive /home/hiseq.clinical/ENCRYPT/*HHH5TDSX2*
Call ['/bin/dsmc', 'q', 'archive', '/home/hiseq.clinical/ENCRYPT/*HHH5TDSX2*'] exit with a non zero exit code

Traceback (most recent call last):
  File "/home/proj/stage/bin/miniconda3/envs/S_main/bin/cg", line 8, in <module>
    sys.exit(base())
  File "/home/proj/stage/bin/miniconda3/envs/S_main/lib/python3.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/proj/stage/bin/miniconda3/envs/S_main/lib/python3.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/proj/stage/bin/miniconda3/envs/S_main/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/proj/stage/bin/miniconda3/envs/S_main/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/proj/stage/bin/miniconda3/envs/S_main/lib/python3.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/proj/stage/bin/miniconda3/envs/S_main/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/proj/stage/bin/miniconda3/envs/S_main/lib/python3.7/site-packages/click/decorators.py", line 27, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/home/proj/stage/bin/miniconda3/envs/S_main/lib/python3.7/site-packages/cg/cli/backup.py", line 63, in fetch_flow_cell
    retrieval_time: Optional[float] = backup_api.fetch_flow_cell(flow_cell=flow_cell)
  File "/home/proj/stage/bin/miniconda3/envs/S_main/lib/python3.7/site-packages/cg/meta/backup/backup.py", line 89, in fetch_flow_cell
    pdc_flow_cell_query: List[str] = self.query_pdc_for_flow_cell(flow_cell.name)
  File "/home/proj/stage/bin/miniconda3/envs/S_main/lib/python3.7/site-packages/cg/meta/backup/backup.py", line 267, in query_pdc_for_flow_cell
    raise error
  File "/home/proj/stage/bin/miniconda3/envs/S_main/lib/python3.7/site-packages/cg/meta/backup/backup.py", line 262, in query_pdc_for_flow_cell
    self.pdc.query_pdc(search_pattern=search_pattern)
  File "/home/proj/stage/bin/miniconda3/envs/S_main/lib/python3.7/site-packages/cg/meta/backup/pdc.py", line 33, in query_pdc
    self.process.run_command(parameters=command)
  File "/home/proj/stage/bin/miniconda3/envs/S_main/lib/python3.7/site-packages/cg/utils/commands.py", line 107, in run_command
    raise CalledProcessError(res.returncode, command)
subprocess.CalledProcessError: Command '['/bin/dsmc', 'q', 'archive', '/home/hiseq.clinical/ENCRYPT/*HHH5TDSX2*']' returned non-zero exit status 12.

@seallard
Copy link
Contributor Author

@Vince-janv and @karlnyr, please have a look at this when you are back. The reanalysis of these two flow cells have been delayed by 1.5 months or so

@beatrizsavinhas
Copy link
Contributor

I believe the same problem is happening with flowcell HL532DSXY as well which is delaying the delivery of a re-run.

@Vince-janv
Copy link
Contributor

I believe the same problem is happening with flowcell HL532DSXY as well which is delaying the delivery of a re-run.

Demultiplexing of HL532DSXY is now running after some manual handling

@Vince-janv
Copy link
Contributor

For future debugging the flow cell had already been decrypted but not untared. I did the following:

  1. untared it with: tar -xf <flow_cell_run_dir>/211029_A00621_0522_AHHH5TDSX2.tar.gz --exclude=RTAComplete.txt --exclude=demuxstarted.txt --exclude=Thumbnail_Images -C <flow_cell_run_dir> (in a tmux session)
  2. touched the RTAComplete.txt
  3. removed the tar and gpg files

These are basically all the operations from this line and onwards in cg.meta.backup.backup.BackupAPI._process_flow_cell

@beatrizsavinhas
Copy link
Contributor

Flow cells HKK23DSXY and H3MG5DSX2 are stuck as "requested" in StatusDB.

We have the following cron error for flow cell HKK23DSXY from 29 Aug 2023, 23:04:

2023-08-29 21:43:33 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.meta.backup.backup[56113] INFO HKK23DSXY: retrieving from PDC
2023-08-29 21:43:33 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.utils.commands[56113] INFO Running command /bin/dsmc q archive /home/hiseq.clinical/ENCRYPT/*HKK23DSXY*
2023-08-29 21:43:36 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.utils.commands[56113] INFO Running command /bin/dsmc q archive /mnt/hds/proj/bioinfo/TO_PDC/*HKK23DSXY*
2023-08-29 21:43:38 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.utils.commands[56113] CRITICAL Call ['/bin/dsmc', 'q', 'archive', '/mnt/hds/proj/bioinfo/TO_PDC/*HKK23DSXY*'] exit with a non zero exit code
2023-08-29 21:43:38 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.utils.commands[56113] CRITICAL ANS1092W No files matching search criteria were found
2023-08-29 21:43:38 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.meta.backup.backup[56113] INFO No archived files found for pdc query ['/home/hiseq.clinical/ENCRYPT/*HKK23DSXY*', '/mnt/hds/proj/bioinfo/TO_PDC/*HKK23DSXY*'], testing legacy directory
2023-08-29 21:43:38 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.meta.backup.backup[56113] INFO Found archived files for pdc queries ['/home/hiseq.clinical/ENCRYPT/*HKK23DSXY*', '/mnt/hds/proj/bioinfo/TO_PDC/*HKK23DSXY*']
2023-08-29 21:43:38 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.meta.backup.backup[56113] INFO Encryption key found: /home/hiseq.clinical/ENCRYPT/210115_A00689_0215_BHKK23DSXY.key.gpg
2023-08-29 21:43:38 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.meta.backup.backup[56113] INFO Flow cell found: /home/hiseq.clinical/ENCRYPT/210115_A00689_0215_BHKK23DSXY.tar.gz.gpg
2023-08-29 21:43:38 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.utils.commands[56113] INFO Running command /bin/dsmc retrieve -replace=yes /home/hiseq.clinical/ENCRYPT/210115_A00689_0215_BHKK23DSXY.key.gpg /home/proj/production/flow_cells/210115_A00689_0215_BHKK23DSXY.key.gpg
2023-08-29 21:43:44 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.utils.commands[56113] CRITICAL Call ['/bin/dsmc', 'retrieve', '-replace=yes', '/home/hiseq.clinical/ENCRYPT/210115_A00689_0215_BHKK23DSXY.key.gpg', '/home/proj/production/flow_cells/210115_A00689_0215_BHKK23DSXY.key.gpg'] exit with a non zero exit code
2023-08-29 21:43:44 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.utils.commands[56113] CRITICAL
2023-08-29 21:43:44 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.meta.backup.backup[56113] WARNING WARNING for retrieval of encryption key of flow cell HKK23DSXY, please check dsmerror.log
2023-08-29 21:43:44 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.utils.commands[56113] INFO Running command /bin/dsmc retrieve -replace=yes /home/hiseq.clinical/ENCRYPT/210115_A00689_0215_BHKK23DSXY.tar.gz.gpg /home/proj/production/flow_cells/210115_A00689_0215_BHKK23DSXY.tar.gz.gpg
2023-08-29 23:04:41 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.utils.commands[56113] CRITICAL Call ['/bin/dsmc', 'retrieve', '-replace=yes', '/home/hiseq.clinical/ENCRYPT/210115_A00689_0215_BHKK23DSXY.tar.gz.gpg', '/home/proj/production/flow_cells/210115_A00689_0215_BHKK23DSXY.tar.gz.gpg'] exit with a non zero exit code
2023-08-29 23:04:41 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.utils.commands[56113] CRITICAL
2023-08-29 23:04:41 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.meta.backup.backup[56113] WARNING WARNING for retrieval of flow cell HKK23DSXY, please check dsmerror.log
2023-08-29 23:04:41 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.meta.backup.backup[56113] INFO Status for flow cell HKK23DSXY set to retrieved
2023-08-29 23:04:41 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.meta.encryption.encryption[56113] INFO Starting GPG command:
2023-08-29 23:04:41 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.meta.encryption.encryption[56113] INFO /bin/gpg --decrypt --batch --cipher-algo AES256 --passphrase Clinical Genomics -o /home/proj/production/flow_cells/210115_A00689_0215_BHKK23DSXY.key /home/proj/production/flow_cells/210115_A00689_0215_BHKK23DSXY.key.gpg
2023-08-29 23:04:41 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.utils.commands[56113] INFO Running command /bin/gpg --decrypt --batch --cipher-algo AES256 --passphrase Clinical Genomics -o /home/proj/production/flow_cells/210115_A00689_0215_BHKK23DSXY.key /home/proj/production/flow_cells/210115_A00689_0215_BHKK23DSXY.key.gpg
2023-08-29 23:04:41 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.utils.commands[56113] CRITICAL Call ['/bin/gpg', '--decrypt', '--batch', '--cipher-algo', <CipherAlgorithm.AES256: 'AES256'>, '--passphrase', <EncryptionUserID.HASTA_USER_ID: 'Clinical Genomics'>, '-o', '/home/proj/production/flow_cells/210115_A00689_0215_BHKK23DSXY.key', '/home/proj/production/flow_cells/210115_A00689_0215_BHKK23DSXY.key.gpg'] exit with a non zero exit code
2023-08-29 23:04:41 [hasta.scilifelab.se](http://hasta.scilifelab.se/) cg.utils.commands[56113] CRITICAL gpg: encrypted with 2048-bit RSA key, ID 39E1F18E, created 2016-03-10

The key seems to be retrieved but it can't proceed after that. Might have something to do with the change in the flow cell runs directory.

@Vince-janv Vince-janv mentioned this issue Sep 12, 2023
Closed
3 tasks
@ChrOertlin
Copy link
Contributor

@henrikstranneheim I think you spent some time on the PDC retrieval, right? Was this resolved?

@Vince-janv
Copy link
Contributor

This has been solved here: #2638
The whole module is still a bit shaky though

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ChrOertlin @beatrizsavinhas @seallard @Vince-janv