Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically sync aggragate datasets to AWS #31

Open
3 tasks
k1o0 opened this issue Jun 16, 2023 · 1 comment
Open
3 tasks

Automatically sync aggragate datasets to AWS #31

k1o0 opened this issue Jun 16, 2023 · 1 comment
Assignees

Comments

@k1o0
Copy link
Contributor

k1o0 commented Jun 16, 2023

The current update_aws management command syncs datasets grouped by session and doesn't support aggregate datasets.

  • Create method in management command to sync base aggregates folder*
  • Create/modify S3 file records for aggregate datasets after sync; delete datasets with only file record on S3
  • Skip session level dataset sync when eid is null (here)

*aws s3 sync /mnt/ibl/aggregates s3://ibl-brain-wide-map-private/aggregates --profile ibladmin --exclude *.log --no-progress --dryrun

@oliche
Copy link
Member

oliche commented Jul 3, 2023

Filter datasets per empty sessions will give us the set of data to sync.
Then need to go over full architecture to register datasets

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants