Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(lib/runtime/storage): Don't rely on trie snapshots for storage transactions #3777

Merged
merged 60 commits into from
Mar 26, 2024
Merged
Show file tree
Hide file tree
Changes from 43 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
a5fb392
refactor(lib/runtime/storage): don't use snapshots
dimartiro Feb 27, 2024
5be7cf1
Merge branch 'development' of https://github.com/ChainSafe/gossamer i…
dimartiro Feb 27, 2024
484d141
chore(license): Add missing license
dimartiro Feb 27, 2024
3c20dae
Fix merge
dimartiro Feb 27, 2024
4154eba
refactor(lib/runtime/storage): rename to storageDiff
dimartiro Mar 1, 2024
a20ecc6
Merge branch 'development' of https://github.com/ChainSafe/gossamer i…
dimartiro Mar 1, 2024
75a2d2b
Merge branch 'development' of https://github.com/ChainSafe/gossamer i…
dimartiro Mar 1, 2024
ba63f5d
Merge branch 'development' into diego/decouple-storage_transaction-ma…
dimartiro Mar 3, 2024
f004255
chore(deps): bump github.com/ethereum/go-ethereum from 1.13.13 to 1.1…
dependabot[bot] Mar 4, 2024
c4566a8
chore(deps): bump github.com/prometheus/client_golang from 1.18.0 to …
dependabot[bot] Mar 5, 2024
a4d14f9
chore(deps): bump github.com/go-playground/validator/v10 from 10.18.0…
dependabot[bot] Mar 5, 2024
c20fc1d
chore(deps): bump github.com/stretchr/testify from 1.8.4 to 1.9.0 (#3…
dependabot[bot] Mar 5, 2024
7419f0e
fix: decode limit as option of uint32 (#3792)
radkomih Mar 6, 2024
4166f1b
chore: add infra label (#3788)
P1sar Mar 7, 2024
9cc9e56
chore(deps): bump golang.org/x/crypto from 0.19.0 to 0.21.0 (#3790)
dependabot[bot] Mar 7, 2024
7f835e9
chore(deps): bump codecov/codecov-action from 4.0.2 to 4.1.0 (#3781)
dependabot[bot] Mar 7, 2024
10f0811
chore(deps): bump google.golang.org/protobuf from 1.32.0 to 1.33.0 (#…
dependabot[bot] Mar 7, 2024
6c76e59
fix(sync): tip sync can follow chain and finalize blocks (#3765)
jimjbrettj Mar 7, 2024
1afb243
Merge branch 'development' of https://github.com/ChainSafe/gossamer i…
dimartiro Mar 11, 2024
1d67c76
Simplify state root method
dimartiro Mar 11, 2024
2e43c0b
Add missing license
dimartiro Mar 11, 2024
1bbf3f6
Complete clear prefix methods
dimartiro Mar 11, 2024
7e2e998
Fix return
dimartiro Mar 11, 2024
0ddfa43
Merge branch 'development' into diego/decouple-storage_transaction-ma…
dimartiro Mar 12, 2024
852b087
Fix all deleted calc
dimartiro Mar 12, 2024
3c0af1e
Add missing implementations
dimartiro Mar 12, 2024
5710c6b
Fix tests
dimartiro Mar 13, 2024
0860594
Remove unnused childTrieEntries method
dimartiro Mar 13, 2024
1959cec
Remove commented code
dimartiro Mar 13, 2024
ff2d255
Rename mutex
dimartiro Mar 13, 2024
8d6dbc7
Adding storagediff doc
dimartiro Mar 13, 2024
ec1b9b9
Add documentation
dimartiro Mar 13, 2024
86c9d35
Undo removed function
dimartiro Mar 13, 2024
00a5f9b
Change constructor name
dimartiro Mar 19, 2024
95db16b
Fix clear prefix
dimartiro Mar 19, 2024
09875b7
Fix delete from child
dimartiro Mar 19, 2024
98e1128
Fix linter
dimartiro Mar 19, 2024
e6b18a8
Fix deleteChildLimit
dimartiro Mar 19, 2024
d333120
Fix lint
dimartiro Mar 19, 2024
cc878ff
Fix ClearPrefixInChildWithLimit and add more tests
dimartiro Mar 20, 2024
a15b1f5
Finish storage diff testing
dimartiro Mar 20, 2024
af8f93e
Linting
dimartiro Mar 20, 2024
133d355
Check nil transaction
dimartiro Mar 20, 2024
57522f3
Unskip test
dimartiro Mar 21, 2024
db9d171
Change comment
dimartiro Mar 21, 2024
a9ed0bf
Break down interfaces
dimartiro Mar 21, 2024
735aefe
Merge branch 'development' into diego/decouple-storage_transaction-ma…
dimartiro Mar 22, 2024
0be1f40
Remove mtx from storagediff
dimartiro Mar 22, 2024
e564823
Merge branch 'diego/decouple-storage_transaction-management' of githu…
dimartiro Mar 22, 2024
f384308
Merge branch 'development' into diego/decouple-storage_transaction-ma…
dimartiro Mar 22, 2024
c333430
Add more tests for new transactions
dimartiro Mar 22, 2024
e138add
Merge branch 'diego/decouple-storage_transaction-management' of githu…
dimartiro Mar 22, 2024
6e0a7fa
Linting
dimartiro Mar 22, 2024
ebbee81
Add more tests for new transactions
dimartiro Mar 22, 2024
1c0a05a
Remove all delete child limit test
dimartiro Mar 22, 2024
5e2bbb4
Complete test suite
dimartiro Mar 22, 2024
fb4919e
Makes ci happy
dimartiro Mar 22, 2024
fb606b0
Unskip test
dimartiro Mar 25, 2024
4cfed65
Merge branch 'development' into diego/decouple-storage_transaction-ma…
dimartiro Mar 25, 2024
fdca5bf
Add todo for future discussion
dimartiro Mar 26, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions dot/rpc/modules/childstate_integration_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -247,12 +247,13 @@ func setupChildStateStorage(t *testing.T) (*ChildStateModule, common.Hash) {
tr.Put([]byte(":first_key"), []byte(":value1"))
tr.Put([]byte(":second_key"), []byte(":second_value"))

childTr := trie.NewEmptyTrie()
childTr.Put([]byte(":child_first"), []byte(":child_first_value"))
childTr.Put([]byte(":child_second"), []byte(":child_second_value"))
childTr.Put([]byte(":another_child"), []byte("value"))
childStorageKey := []byte(":child_storage_key")

err = tr.SetChild([]byte(":child_storage_key"), childTr)
err = tr.SetChildStorage(childStorageKey, []byte(":child_first"), []byte(":child_first_value"))
require.NoError(t, err)
err = tr.SetChildStorage(childStorageKey, []byte(":child_second"), []byte(":child_second_value"))
require.NoError(t, err)
err = tr.SetChildStorage(childStorageKey, []byte(":another_child"), []byte("value"))
require.NoError(t, err)

stateRoot, err := tr.Root()
Expand Down
11 changes: 6 additions & 5 deletions dot/rpc/modules/childstate_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,13 @@ func createTestTrieState(t *testing.T) (*trie.Trie, common.Hash) {
tr.Put([]byte(":first_key"), []byte(":value1"))
tr.Put([]byte(":second_key"), []byte(":second_value"))

childTr := trie.NewEmptyTrie()
childTr.Put([]byte(":child_first"), []byte(":child_first_value"))
childTr.Put([]byte(":child_second"), []byte(":child_second_value"))
childTr.Put([]byte(":another_child"), []byte("value"))
childStorageKey := []byte(":child_storage_key")

err := tr.SetChild([]byte(":child_storage_key"), childTr)
err := tr.SetChildStorage(childStorageKey, []byte(":child_first"), []byte(":child_first_value"))
require.NoError(t, err)
err = tr.SetChildStorage(childStorageKey, []byte(":child_second"), []byte(":child_second_value"))
require.NoError(t, err)
err = tr.SetChildStorage(childStorageKey, []byte(":another_child"), []byte("value"))
require.NoError(t, err)

stateRoot, err := tr.Root()
Expand Down
19 changes: 6 additions & 13 deletions dot/state/storage_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@ import (
"github.com/ChainSafe/gossamer/lib/common"
runtime "github.com/ChainSafe/gossamer/lib/runtime/storage"
"github.com/ChainSafe/gossamer/pkg/trie"
"github.com/ChainSafe/gossamer/pkg/trie/node"
"go.uber.org/mock/gomock"

"github.com/stretchr/testify/require"
Expand All @@ -31,6 +30,8 @@ func newTestStorageState(t *testing.T) *StorageState {
}

func TestStorage_StoreAndLoadTrie(t *testing.T) {
t.Skip()
dimartiro marked this conversation as resolved.
Show resolved Hide resolved

storage := newTestStorageState(t)
ts, err := storage.TrieState(&trie.EmptyHash)
require.NoError(t, err)
Expand All @@ -44,10 +45,9 @@ func TestStorage_StoreAndLoadTrie(t *testing.T) {

trie, err := storage.LoadFromDB(root)
require.NoError(t, err)
ts2 := runtime.NewTrieState(trie)
newSnapshot := ts2.Snapshot()
ts2 := runtime.NewTrieState(trie).Trie()

require.True(t, ts.Trie().Equal(newSnapshot))
require.True(t, ts.Trie().Equal(ts2))
}

func TestStorage_GetStorageByBlockHash(t *testing.T) {
Expand Down Expand Up @@ -183,16 +183,9 @@ func TestGetStorageChildAndGetStorageFromChild(t *testing.T) {
trieDB := NewMockDatabase(ctrl)
trieDB.EXPECT().Get(gomock.Any()).Times(0)

trieRoot := &node.Node{
PartialKey: []byte{1, 2},
StorageValue: []byte{3, 4},
Dirty: true,
}
testChildTrie := trie.NewTrie(trieRoot, trieDB)

testChildTrie.Put([]byte("keyInsidechild"), []byte("voila"))
genTrie.PutIntoChild([]byte("keyToChild"), []byte{1, 2}, []byte{3, 4})
genTrie.PutIntoChild([]byte("keyToChild"), []byte("keyInsidechild"), []byte("voila"))

err = genTrie.SetChild([]byte("keyToChild"), testChildTrie)
require.NoError(t, err)

tries := newTriesEmpty()
Expand Down
22 changes: 14 additions & 8 deletions lib/runtime/interfaces.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,28 +11,34 @@ import (

// Storage runtime interface.
type Storage interface {
// Main trie
jimjbrettj marked this conversation as resolved.
Show resolved Hide resolved
dimartiro marked this conversation as resolved.
Show resolved Hide resolved
Root() (common.Hash, error)
Put(key []byte, value []byte) (err error)
Get(key []byte) []byte
Root() (common.Hash, error)
SetChild(keyToChild []byte, child *trie.Trie) error
Delete(key []byte) (err error)
NextKey([]byte) []byte
ClearPrefix(prefix []byte) (err error)
ClearPrefixLimit(prefix []byte, limit uint32) (
deleted uint32, allDeleted bool, err error)

// Child tries
GetChildRoot(keyToChild []byte) (common.Hash, error)
SetChildStorage(keyToChild, key, value []byte) error
GetChildStorage(keyToChild, key []byte) ([]byte, error)
Delete(key []byte) (err error)
DeleteChild(keyToChild []byte) (err error)
DeleteChildLimit(keyToChild []byte, limit *[]byte) (
deleted uint32, allDeleted bool, err error)
ClearChildStorage(keyToChild, key []byte) error
NextKey([]byte) []byte
ClearPrefixInChild(keyToChild, prefix []byte) error
ClearPrefixInChildWithLimit(keyToChild, prefix []byte, limit uint32) (uint32, bool, error)
GetChildNextKey(keyToChild, key []byte) ([]byte, error)
GetChild(keyToChild []byte) (*trie.Trie, error)
ClearPrefix(prefix []byte) (err error)
ClearPrefixLimit(prefix []byte, limit uint32) (
deleted uint32, allDeleted bool, err error)

// Transactions
StartTransaction()
CommitTransaction()
RollbackTransaction()

// Runtime
LoadCode() []byte
SetVersion(v trie.TrieLayout)
}
Expand Down
Loading
Loading