Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate upload to / download from S3 #28

Open
joshmoore opened this issue Apr 22, 2021 · 0 comments
Open

Automate upload to / download from S3 #28

joshmoore opened this issue Apr 22, 2021 · 0 comments

Comments

@joshmoore
Copy link
Member

Currently the most manual part of the process is migrating files to and from S3. In the docker case, this happens mostly automatically. On AWS, however, it requires several manual steps which become increasingly time consuming with growing file size.

A background process and/or task manager (prefect?) would be ideal to handle these migrations:

  • Create two AWS instances ("A" and "B")
  • Generate all files on instance A
  • Upload all files to S3 from instance A, measuring the upload speeds
  • Download all files from S3 to instance B, measuring the download speeds
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant