Consider backing off if RequestBroadcast errors #632
Comments
|
2024-09-09: Discussed in the daily sync that it is lower on the priority list and therefore I will not assign |
masih
added a commit
that referenced
this issue
Oct 9, 2024
GPBFT presciently ignores broadcast and rebroadcast requests as it is beyond its boundaries of responsibility to do something about. In practice such failures may be a sign that pubsub is overwhelmed with messages. Therefore, ideally the system should avoid aggravating the situation by requesting further broadcasts. This is specially important in re-broadcast requests because it often involves batch message publication. The changes here wrap the pubsub publication calls with a circuit breaker that will open on consecutive errors (set to `5`), and will not attempt to actually publish messages until a reset timeout (set to `3s`) has passed. Fixes #632
masih
added a commit
that referenced
this issue
Oct 9, 2024
GPBFT silently ignores broadcast and rebroadcast requests as it is beyond its boundaries of responsibility to do something about. In practice such failures may be a sign that pubsub is overwhelmed with messages. Therefore, ideally the system should avoid aggravating the situation by requesting further broadcasts. This is specially important in re-broadcast requests because it often involves batch message publication. The changes here wrap the pubsub publication calls with a circuit breaker that will open on consecutive errors (set to `5`), and will not attempt to actually publish messages until a reset timeout (set to `3s`) has passed. Fixes #632
masih
modified the milestones:
Milestone 2.5: Mainnet Deployment Readiness,
Milestone 3: Exchange and Ecosystem Integration
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The GPBFT implementation here silently logs errors that occur as a result of requesting broadcast.
In practice, if such situations occur it may be better to backoff from sending consecutive requests, e.g. that may be caused by pubsub congestion.
Using a circuit breaker pattern at host implementation in
f3package might just do the trick without the need for larger refactor across the coregpbftpackage/protocol changes.Originally posted by @Stebalien in #627 (comment)
The text was updated successfully, but these errors were encountered: