You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We currently default to never evicting inputs from ProxyStore, which has pros and cons.
The main con is that if you run a large number of tasks, the data store will become full of data that is no longer useful. This is a big problem if that data is stored in an Redis store in RAM. We automatically evict results from the store for this reason.
The pro is that we avoid making it harder to re-use the same input between multiple tasks and make it so that workflow engines can restart tasks without worry.
We should make it easier for users to choose their own behavior and, perhaps, make it so Colmena only autoevicts an input from a store after a task completes?
The text was updated successfully, but these errors were encountered:
What we could do is store the keys of the proxies created automatically when task inputs are serialized, and have the queue class automatically evict them when a result is received and no further tasks are outstanding with those objects
We currently default to never evicting inputs from ProxyStore, which has pros and cons.
The main con is that if you run a large number of tasks, the data store will become full of data that is no longer useful. This is a big problem if that data is stored in an Redis store in RAM. We automatically evict results from the store for this reason.
The pro is that we avoid making it harder to re-use the same input between multiple tasks and make it so that workflow engines can restart tasks without worry.
We should make it easier for users to choose their own behavior and, perhaps, make it so Colmena only autoevicts an input from a store after a task completes?
The text was updated successfully, but these errors were encountered: