Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Separate Download Bool for images and workspace #193

Open
ErinWeisbart opened this issue Feb 5, 2025 · 1 comment
Open

Separate Download Bool for images and workspace #193

ErinWeisbart opened this issue Feb 5, 2025 · 1 comment

Comments

@ErinWeisbart
Copy link
Member

ErinWeisbart commented Feb 5, 2025

Might want to mount one bucket but download from the other (SOURCE_BUCKET vs WORKSPACE_BUCKET).
(e.g. reading images off of the Cell Painting Gallery, mount is good to prevent file download fees but workspace files are likely in another bucket)

We currently only support:
Same bucket - Download = True for both
Same bucket - Download = False for both

I think we want to support:
Different bucket - Download = True for both
Different bucket - Download = False for source, Download = True for workspace

I think we don't need to support:
Different bucket - Download = False for both (i.e. mounting two buckets)

I'm indifferent to supporting:
Different bucket - Download = True for source, Download = False for workspace

@ErinWeisbart
Copy link
Member Author

I'm leaning toward:

  1. we have mounting for just the SOURCE_BUCKET (where the images are) and default to download workspace files regardless of download bool. Pro, doesn't expand the config, easily adds on support for the two additional options I think we should support. Con, might be situations with large workspace files that we really would want to mount if we're getting more into DL stuff?? If so, could likely workaround by making custom docker as likely large workspace files would be consistent between batches.

Possible that instead:
2) we separate the bool for mount of different buckets. Pro, more configurable. Con, would mean we would need to support mounting two buckets as I really don't want a situation where someone could set something in the config that we don't support

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant