Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reject all futures once a FutureStore is closed. #201

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

isra17
Copy link

@isra17 isra17 commented Aug 15, 2024

This fix a race condition where a writer_drain future might be created after a disconnection and the FutureStore had all the future rejected. In this case, the new future won't ever be completed since the connection is not writing anymore.

I have seen it happen in production on high concurrency publish, where disconnect can happen while a task is waiting for the publish lock, then after everything get rejected and the lock is released, the waiting task get the lock, send data and wait for the drain that will never come.

An alternative fix would be to add is_closed checks everywhere before we create future, but my current solution seems to be foolproof. The FutureStore is never reopened, so once it's close it should stay close and fail any attempt at creating new futures.

I also added some extra check so the test behavior doesn't change (Otherwise the CancelledError get raised instead of the expected InvalidChannelState)

I'm quickly opening this PR so it can get reviewed, but I will add a test by tomorrow.

This fix a race condition where a writer_drain future might be created
after a disconnection and the FutureStore had all the future rejected.
In this case, the new future won't ever be completed since the
connection is not writing anymore.
@isra17
Copy link
Author

isra17 commented Aug 16, 2024

Added a test that fail in master and pass in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant