fix: (DutyScheduler) duties reset race condition #1741
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
y0sher
reviewed
Sep 16, 2024
| d.m[period][validatorIndex] = dutyDescriptor[eth2apiv1.SyncCommitteeDuty]{ | ||
| duty: duty, | ||
| inCommittee: inCommittee, | ||
| func (d *SyncCommitteeDuties) Set(period uint64, duties []DutyDescriptor[eth2apiv1.SyncCommitteeDuty]) { |
There was a problem hiding this comment.
you removed the mutex lock here
There was a problem hiding this comment.
@y0sher good catch, fixed
i also pushed a bunch of refactors
operator/duties/dutystore/duties.go
Outdated
Comment on lines
82
to
87
| d.m[epoch] = make(map[phase0.Slot]map[phase0.ValidatorIndex]StoreDuty[D]) | ||
| for _, duty := range duties { | ||
| if _, ok := d.m[epoch][duty.Slot]; !ok { | ||
| d.m[epoch][duty.Slot] = make(map[phase0.ValidatorIndex]StoreDuty[D]) | ||
| } | ||
| d.m[epoch][duty.Slot][duty.ValidatorIndex] = duty |
There was a problem hiding this comment.
we can reduce lock time and reduce redundant lookups
func (d *Duties[D]) Set(epoch phase0.Epoch, duties []StoreDuty[D]) {
newDuties := make(map[phase0.Slot]map[phase0.ValidatorIndex]StoreDuty[D])
for _, duty := range duties {
if _, slotExists := newDuties[duty.Slot]; !slotExists {
newDuties[duty.Slot] = make(map[phase0.ValidatorIndex]StoreDuty[D])
}
newDuties[duty.Slot][duty.ValidatorIndex] = duty
}
d.mu.Lock()
defer d.mu.Unlock()
d.m[epoch] = newDuties
}There was a problem hiding this comment.
@olegshmuelov nice optimization, applied both 👌
Comment on lines
63
to
71
| func (d *SyncCommitteeDuties) Set(period uint64, duties []StoreSyncCommitteeDuty) { | ||
| d.mu.Lock() | ||
| defer d.mu.Unlock() | ||
|
|
||
| if _, ok := d.m[period]; !ok { | ||
| d.m[period] = make(map[phase0.ValidatorIndex]dutyDescriptor[eth2apiv1.SyncCommitteeDuty]) | ||
| } | ||
|
|
||
| d.m[period][validatorIndex] = dutyDescriptor[eth2apiv1.SyncCommitteeDuty]{ | ||
| duty: duty, | ||
| inCommittee: inCommittee, | ||
| d.m[period] = make(map[phase0.ValidatorIndex]StoreSyncCommitteeDuty) | ||
| for _, duty := range duties { | ||
| d.m[period][duty.ValidatorIndex] = duty | ||
| } | ||
| } |
There was a problem hiding this comment.
same here
func (d *SyncCommitteeDuties) Set(period uint64, duties []StoreSyncCommitteeDuty) {
newDuties := make(map[phase0.ValidatorIndex]StoreSyncCommitteeDuty, len(duties))
for _, duty := range duties {
newDuties[duty.ValidatorIndex] = duty
}
d.mu.Lock()
defer d.mu.Unlock()
d.m[period] = newDuties
}
y0sher
approved these changes
Sep 17, 2024
olegshmuelov
approved these changes
Sep 17, 2024
This was referenced Sep 18, 2024
Open
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes a race condition discovered by @olegshmuelov which happens right after an indices changes and re-orgs, when AttesterHandler and SyncCommitteeHandler reset their respective duty maps prior to re-fetching, during which any reads by CommitteeHandler would end up with zero duties.
The fix changes how we populate the map, opting for
fetch -> setrather thanreset -> fetch -> add.This bug was first fixed in #1725 using a different approach. Since that PR contains other other improvements & fixes, then this PR doesn't close it, and we should combine both after verifying this approach works.