Reject all futures once a FutureStore is closed. #201
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fix a race condition where a writer_drain future might be created after a disconnection and the FutureStore had all the future rejected. In this case, the new future won't ever be completed since the connection is not writing anymore.
I have seen it happen in production on high concurrency publish, where disconnect can happen while a task is waiting for the publish
lock
, then after everything get rejected and the lock is released, the waiting task get the lock, send data and wait for the drain that will never come.An alternative fix would be to add
is_closed
checks everywhere before we create future, but my current solution seems to be foolproof. TheFutureStore
is never reopened, so once it's close it should stay close and fail any attempt at creating new futures.I also added some extra check so the test behavior doesn't change (Otherwise the
CancelledError
get raised instead of the expectedInvalidChannelState
)I'm quickly opening this PR so it can get reviewed, but I will add a test by tomorrow.