Skip to content

feat: support IAVL v2 #24960

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 18 commits into
base: main
Choose a base branch
from
Draft

feat: support IAVL v2 #24960

wants to merge 18 commits into from

Conversation

aaronc
Copy link
Member

@aaronc aaronc commented Jul 21, 2025

Description

Closes: #XXXX

github-advanced-security[bot]
github-advanced-security bot found potential problems Jul 21, 2025
View reviewed changes
store/iavl2/store.go
defer st.metrics.MeasureSince("store", "iavl", "get")
value, err := st.tree.Get(key)
if err != nil {
panic(err)

Check warning

Code scanning / CodeQL

Panic in BeginBock or EndBlock consensus methods Warning

Possible panics in BeginBock- or EndBlock-related consensus methods could cause a chain halt
store/iavl2/store.go
defer st.metrics.MeasureSince("store", "iavl", "has")
has, err := st.tree.Has(key)
if err != nil {
panic(err)

Check warning

Code scanning / CodeQL

Panic in BeginBock or EndBlock consensus methods Warning

Possible panics in BeginBock- or EndBlock-related consensus methods could cause a chain halt
store/iavl2/store.go
defer st.metrics.MeasureSince("store", "iavl", "delete")
_, _, err := st.tree.Remove(key)
if err != nil {
panic(err)

Check warning

Code scanning / CodeQL

Panic in BeginBock or EndBlock consensus methods Warning

Possible panics in BeginBock- or EndBlock-related consensus methods could cause a chain halt
store/iavl2/store.go
func (st *Store) Iterator(start, end []byte) types.Iterator {
iterator, err := st.tree.Iterator(start, end, true)
if err != nil {
panic(err)

Check warning

Code scanning / CodeQL

Panic in BeginBock or EndBlock consensus methods Warning

Possible panics in BeginBock- or EndBlock-related consensus methods could cause a chain halt
store/iavl2/store.go
func (st *Store) ReverseIterator(start, end []byte) types.Iterator {
iterator, err := st.tree.Iterator(start, end, false)
if err != nil {
panic(err)

Check warning

Code scanning / CodeQL

Panic in BeginBock or EndBlock consensus methods Warning

Possible panics in BeginBock- or EndBlock-related consensus methods could cause a chain halt
@aaronc
Copy link
Member Author

aaronc commented Jul 21, 2025 •
edited
Loading

So, it seems like we're running into an issue with iterators in IAVL v2. To test this, I have done two runs of the TestStakeUnstake system test, one with IAVL v2 and one without and added logging statements in validator.go to capture what's happening with iteration right before the error.

Without regular IAVL, we always see this in the logs:

1:25PM INF iterating validator power keys key=111402cefc46c0a393c14d1b7d74dabfa42eb574d5fe module=x/staking value=0864
1:25PM INF iterating validator power keys key=111412886e3fea32ff4aba66c4f624076751ee1ca802 module=x/staking value=0864
1:25PM INF iterating validator power keys key=111423c714179cc5102c03002c884dd635cc652c30dc module=x/staking value=0864
1:25PM INF iterating validator power keys key=11146ca69686b6ba1a4c72af3f1674743ccf7fa35dd3 module=x/staking value=0864

With IAVL v2, the first time this code runs, it looks the same as above, but the second time we see this and an exception trace:

1:29PM INF iterating validator power keys key=111433746e4e3d2cc8287cc8448ea35779146831f139 module=x/staking value=0864
1:29PM INF iterating validator power keys key=111480b24781e33a6b1c257beaa558fc8b7f5fae015d module=x/staking value=0864
1:29PM INF iterating validator power keys key=1114c39962ee869ba237a1e77f133e630e5e50f78aac module=x/staking value=0864
1:29PM INF iterating validator power keys key=1114fd8647c5677aa5abef428c50852d839dac117d1a module=x/staking value=0864
1:29PM INF iterating validator power keys key=12 module=x/staking value=0a03343030
panic: expected key of length at least 3, got 1

goroutine 403 [running]:
github.com/cosmos/cosmos-sdk/types/kv.AssertKeyAtLeastLength(...)
	github.com/cosmos/[email protected]/types/kv/helpers.go:8
github.com/cosmos/cosmos-sdk/x/staking/types.AddressFromLastValidatorPowerKey({0xc00570b208?, 0x1, 0xc003df9058?})
	github.com/cosmos/[email protected]/x/staking/types/keys.go:109 +0xa9
github.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.GetLastValidators({{0x10e1a2c40, 0xc0023a69c0}, {0x10e215e00, 0xc001fdd390}, {0x1579ac5a0, 0xc002b1b020}, {0x1579ac5e8, 0xc002f38360}, {0x10e2161f0, 0xc001d03ae8}, ...}, ...)
	github.com/cosmos/[email protected]/x/staking/keeper/validator.go:443 +0x5e5
github.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.TrackHistoricalInfo({{0x10e1a2c40, 0xc0023a69c0}, {0x10e215e00, 0xc001fdd390}, {0x1579ac5a0, 0xc002b1b020}, {0x1579ac5e8, 0xc002f38360}, {0x10e2161f0, 0xc001d03ae8}, ...}, ...)
	github.com/cosmos/[email protected]/x/staking/keeper/historical_info.go:120 +0x538
github.com/cosmos/cosmos-sdk/x/staking/keeper.(*Keeper).BeginBlocker(0xc001f66100?, {0x10e1f5760?, 0xc0052bb188?})
	github.com/cosmos/[email protected]/x/staking/keeper/abci.go:16 +0x11f
github.com/cosmos/cosmos-sdk/x/staking.AppModule.BeginBlock(...)
	github.com/cosmos/[email protected]/x/staking/module.go:177
github.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x10e1f5808, 0x110726b80}, {0x10e216b40, 0xc001a93380}, {{0x0, 0x0}, {0xc001f646a0, 0x7}, 0x2, ...}, ...})
	github.com/cosmos/[email protected]/types/module/module.go:779 +0x17c
github.com/cosmos/cosmos-sdk/runtime.(*App).BeginBlocker(...)
	github.com/cosmos/[email protected]/runtime/app.go:166
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).beginBlock(0xc002d3b8c8, 0xc000f4a240?)
	github.com/cosmos/[email protected]/baseapp/baseapp.go:677 +0xbf
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).internalFinalizeBlock(0xc002d3b8c8, {0x10e1f5ef0, 0xc001ea44b0}, 0xc000f4a240)
	github.com/cosmos/[email protected]/baseapp/abci.go:791 +0xf14
github.com/cosmos/cosmos-sdk/baseapp/oe.(*OptimisticExecution).Execute.func1()
	github.com/cosmos/[email protected]/baseapp/oe/optimistic_execution.go:108 +0x87
created by github.com/cosmos/cosmos-sdk/baseapp/oe.(*OptimisticExecution).Execute in goroutine 273
	github.com/cosmos/[email protected]/baseapp/oe/optimistic_execution.go:106 +0x52f

It appears that IAVL v2 iterator is going past its stop point and iterating into the key 0x12 even though the prefix key is for 0x11. Looking at https://github.com/cosmos/iavl/blob/master/v2/iterator.go, I see several TODOs. So maybe this iterator code was never actually completed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

C:Store C:x/staking Type: Build

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@aaronc