feat(Nanopore) Check manifest file to confirm data transfer #2791
Conversation
| """Checks if all relevant files from an Illumina sequencing run have been transferred.""" | ||
| for source_flow_cell in Path(source_directory).iterdir(): | ||
| target_flow_cell = Path(target_directory, source_flow_cell.name) | ||
| if is_flow_cell_sync_confirmed(target_flow_cell): | ||
| LOG.debug(f"Flow cell {source_flow_cell} has already been confirmed, skipping.") | ||
| continue | ||
| if is_syncing_complete( | ||
| source_directory=source_flow_cell, | ||
| target_directory=Path(target_directory, source_flow_cell.name), | ||
| ): | ||
| create_copy_complete_file(target_flow_cell) |
There was a problem hiding this comment.
Moved logic from CLI-command to here. This is old code
| @@ -257,6 +257,7 @@ class CGConfig(BaseModel): | |||
| environment: Literal["production", "stage"] = "stage" | |||
| flow_cells_dir: str | |||
| madeline_exe: str | |||
| nanopore_data_directory: str | |||
There was a problem hiding this comment.
| nanopore_data_directory: str | |
| nanopore_flow_cells_dir: str |
The code above makes the distinction between illumina_flow_cells and nanopore_flow_cells Maybe we can keep that theme here as well. It would also give a better context than nanopore_data
There was a problem hiding this comment.
So the structure is like this {experiment_id}/{sample_id}/{start_time}_{device_ID}_{flow_cell_id}_{short_protocol_run_id} where the flow cell is three levels down. I thought calling it flow_cell directory could be misleading.
There was a problem hiding this comment.
Will leave this as is, and revise the naming after the folder restructure
| @@ -257,6 +257,7 @@ class CGConfig(BaseModel): | |||
| environment: Literal["production", "stage"] = "stage" | |||
| flow_cells_dir: str | |||
There was a problem hiding this comment.
| flow_cells_dir: str | |
| illumina_flow_cells_dir: str |
perhaps also?
There was a problem hiding this comment.
Do you think this should be illumina_data_directory?
There was a problem hiding this comment.
I guess it makes sense with the proposed changes in the issue you set up. having both directories under the "sequencing_data" umbrella. Then they could also simply be "illumina" and "nanopore". I don't mind one way or another, just a unified way representing raw sequencing output data would be nice I think
There was a problem hiding this comment.
Agreed, will change it when we've decided on the folder structure 👍
Co-authored-by: ChristianOertlin <[email protected]> Co-authored-by: Henrik Stranneheim <[email protected]>
|
|
@Vince-janv Still relevant? |
Yes. This will be needed if IT does not come up with a inotify solution. Waiting to merge in the event that happens. If I complete the database models and post-processing before hearing anything from them I will merge this |
|
Closing as stale |
Description
This PR uses the created manifest file to check that the transfer is complete and triggers the next step in the automation by touching a file in a directory.
Added
Changed
confirm_transfer_of_illumina_flow_cellThis version is a