-
Notifications
You must be signed in to change notification settings - Fork 112
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
hotfix(message/validation): optimize signer state memory usage #1874
Open
nkryuchkov
wants to merge
23
commits into
stage
Choose a base branch
from
msgvalidation-optimize-signer-state
base: stage
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 19 commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
c8c925c
don't allocate SeenSigners if not needed
nkryuchkov 9f703dd
use SignersBitMask as SeenSigners key
nkryuchkov 55c7525
store full data hash instead of full data
nkryuchkov 8fbb447
compress MessageCounts from 48 bytes to 1
nkryuchkov 5e15f09
fix file name
nkryuchkov 8a92a27
get rid of consensusID
nkryuchkov 9dc78fe
add comments for SeenSigners
nkryuchkov 331d490
use slice instead of map in consensusState
nkryuchkov 1ae8380
fix panic message
nkryuchkov 4d47d4d
refactor getting signer index in committee
nkryuchkov a658aaf
clarify ErrDuplicatedMessage
nkryuchkov 5dd47ea
fix linter
nkryuchkov 203b9be
get rid of panic
nkryuchkov 4d9e546
fix imports
nkryuchkov dc7f0d6
reduce amount of stored slots
nkryuchkov 66e970a
change RWMutex to Mutex
nkryuchkov 6dc13bc
rename states
nkryuchkov 91c43e1
use ttlcache for state
nkryuchkov 055ac8a
fix linter
nkryuchkov 6bb4f65
Merge branch 'stage' into msgvalidation-optimize-signer-state
nkryuchkov a31ac82
Merge branch 'stage' into msgvalidation-optimize-signer-state
nkryuchkov 26c9bc1
fix issues after merging
nkryuchkov 93bc566
remove mutexes in ValidatorState and OperatorState
nkryuchkov File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
package validation | ||
|
||
import ( | ||
"github.com/attestantio/go-eth2-client/spec/phase0" | ||
spectypes "github.com/ssvlabs/ssv-spec/types" | ||
) | ||
|
||
type CommitteeInfo struct { | ||
committeeID spectypes.CommitteeID | ||
committee []spectypes.OperatorID | ||
signerIndices map[spectypes.OperatorID]int | ||
validatorIndices []phase0.ValidatorIndex | ||
} | ||
|
||
func newCommitteeInfo( | ||
committeeID spectypes.CommitteeID, | ||
operators []spectypes.OperatorID, | ||
validatorIndices []phase0.ValidatorIndex, | ||
) CommitteeInfo { | ||
signerIndices := make(map[spectypes.OperatorID]int) | ||
for i, operator := range operators { | ||
signerIndices[operator] = i | ||
} | ||
|
||
return CommitteeInfo{ | ||
committeeID: committeeID, | ||
committee: operators, | ||
signerIndices: signerIndices, | ||
validatorIndices: validatorIndices, | ||
} | ||
} | ||
|
||
// keeping the method for readability and the comment | ||
func (ci *CommitteeInfo) signerIndex(signer spectypes.OperatorID) int { | ||
return ci.signerIndices[signer] // existence must be checked by ErrSignerNotInCommittee | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Was there a specific reason for replacing the RWMutex with a write-only Mutex?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@oleg-ssvlabs
RWMutex
consumes more memory, and theOperatorState
memory consumption seems to be a bottleneck in the exporter. I'd useRWMutex
here only if we benchmark it and see a significant improvementThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ah, interesting. Do you happen to have any numbers for comparison? It would be really compelling to see the difference (specifically between
mutex
andrwMutex
)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's 64 vs 80 bytes for each
OperatorState
IIRC, not much of a difference, but I just wanted to squeeze everything out of the state structure because we allocate a lot of them: for each validator for each role for each operator. So I guess on mainnet the total difference would be a few tens of megabytes (~60K validators * 4 roles * ~5-6 avg committee size * 64 vs 80), which is not very much.I agree that
RWMutex
would reduce mutex block time but I think the difference wouldn't be very big. But generally it looks to me as a trade-off and since we're currently fighting with exporter memory issues, I tend to prefer to reduce the memory useThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@oleg-ssvlabs actually, perhaps we could remove this mutex as @moshe-blox suggested in #2034 (comment). We have validation lock by msg ID, and the message validation doesn't have concurrent checks, so we shouldn't have any data race in
OperatorState
andValidatorState