shutdown: Select on listener on all channel sends #193
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
carlaKC
force-pushed
the
select-send
branch
3 times, most recently
from
11f313b
to
61feaf8
Compare
|
Looking into the build error, can't reproduce locally. |
I think it may be a bug upstream with ref: time-rs/time#693 |
Indeed! Didn't break with my version of rustc - will rebsae this once #195 is in! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ran into a deadlock on shutdown when running with > 100 nodes. Cause was multiple
sendcalls trying to pass events on toreceiverchannels that had already exited (becauselistenersignaled that they should), example here.This is difficult to hit with smaller networks, because most of the time everything just exits with the
listenersignal - it's only when we're in the middle of an operation in multiple tasks where we hit this.This PR takes the approach of just selecting on sender/listener to exit on the sender side. An alternative approach would be to use
receiver.close()and drain the channel. This is a little more boilerplate to update all of our receiver sites, so I went with the sender approach.Tracing via tokio-console is also added behind a development flag, so that debugging issues like this is easier in the future (it has gorgeous dashboards 😻 ).
Related to #162.