You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the most manual part of the process is migrating files to and from S3. In the docker case, this happens mostly automatically. On AWS, however, it requires several manual steps which become increasingly time consuming with growing file size.
A background process and/or task manager (prefect?) would be ideal to handle these migrations:
Create two AWS instances ("A" and "B")
Generate all files on instance A
Upload all files to S3 from instance A, measuring the upload speeds
Download all files from S3 to instance B, measuring the download speeds
The text was updated successfully, but these errors were encountered:
Currently the most manual part of the process is migrating files to and from S3. In the docker case, this happens mostly automatically. On AWS, however, it requires several manual steps which become increasingly time consuming with growing file size.
A background process and/or task manager (prefect?) would be ideal to handle these migrations:
The text was updated successfully, but these errors were encountered: