Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better configuration around whether inputs are autoevicted from ProxyStore #50

Open
WardLT opened this issue Feb 8, 2022 · 2 comments

Comments

@WardLT
Copy link
Collaborator

WardLT commented Feb 8, 2022

We currently default to never evicting inputs from ProxyStore, which has pros and cons.

The main con is that if you run a large number of tasks, the data store will become full of data that is no longer useful. This is a big problem if that data is stored in an Redis store in RAM. We automatically evict results from the store for this reason.

The pro is that we avoid making it harder to re-use the same input between multiple tasks and make it so that workflow engines can restart tasks without worry.

We should make it easier for users to choose their own behavior and, perhaps, make it so Colmena only autoevicts an input from a store after a task completes?

@WardLT
Copy link
Collaborator Author

WardLT commented Sep 20, 2022

What we could do is store the keys of the proxies created automatically when task inputs are serialized, and have the queue class automatically evict them when a result is received and no further tasks are outstanding with those objects

@gpauloski
Copy link
Collaborator

That seems reasonable! The data structure will need to be thread safe (right?) but it otherwise should be pretty lightweight.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants