Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Archive Node Online Migration #1863

Merged
merged 31 commits into from
Oct 30, 2024
Merged

Archive Node Online Migration #1863

merged 31 commits into from
Oct 30, 2024

Conversation

Kbhat1
Copy link
Contributor

@Kbhat1 Kbhat1 commented Sep 18, 2024

Describe your changes and provide context

Testing performed to validate your change

  • Verifying on migration node

tools/migration/ss/migrator.go Fixed Show fixed Hide fixed
tools/migration/ss/migrator.go Fixed Show fixed Hide fixed
return fmt.Errorf("failed to create iterator: %w", err)
}
defer itr.Close()

startTimeBatch := time.Now() // Measure time for every 10,000 iterations

Check warning

Code scanning / CodeQL

Calling the system time Warning

Calling the system time may be a possible source of non-determinism
tools/migration/ss/migrator.go Fixed Show fixed Hide fixed
cmd/seid/cmd/root.go Outdated Show resolved Hide resolved
Kbhat1 and others added 2 commits September 24, 2024 23:35
* Add QMS for online migration

* Fix lint

---------

Co-authored-by: kbhat1 <[email protected]>
cmd/seid/cmd/root.go Fixed Show fixed Hide fixed
Comment on lines +311 to +319
go func() {
homeDir := cast.ToString(appOpts.Get(flags.FlagHome))
stateStore := app.GetStateStore()
migrationHeight := cast.ToInt64(appOpts.Get("migrate-height"))
migrator := ss.NewMigrator(homeDir, db, stateStore)
if err := migrator.Migrate(migrationHeight, homeDir); err != nil {
panic(err)
}
}()

Check notice

Code scanning / CodeQL

Spawning a Go routine Note

Spawning a Go routine may be a possible source of non-determinism
Copy link

codecov bot commented Sep 25, 2024

Codecov Report

Attention: Patch coverage is 30.76923% with 18 lines in your changes missing coverage. Please review.

Project coverage is 61.35%. Comparing base (97346a2) to head (ec3d810).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
app/test_state_store.go 0.00% 12 Missing ⚠️
app/seidb.go 54.54% 4 Missing and 1 partial ⚠️
app/app.go 66.66% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1863      +/-   ##
==========================================
- Coverage   61.56%   61.35%   -0.22%     
==========================================
  Files         263      263              
  Lines       23308    23328      +20     
==========================================
- Hits        14349    14312      -37     
- Misses       7952     8013      +61     
+ Partials     1007     1003       -4     
Files with missing lines Coverage Δ
app/app.go 66.32% <66.66%> (-0.03%) ⬇️
app/seidb.go 80.35% <54.54%> (-7.65%) ⬇️
app/test_state_store.go 56.14% <0.00%> (-2.91%) ⬇️

... and 4 files with indirect coverage changes

tools/migration/ss/migrator.go Fixed Show fixed Hide fixed
if module != startModule && startModule != "" {
continue
}
startTimeModule := time.Now() // Measure time for each module

Check warning

Code scanning / CodeQL

Calling the system time Warning

Calling the system time may be a possible source of non-determinism
})

batchLeafNodeCount = 0
startTimeBatch = time.Now()

Check warning

Code scanning / CodeQL

Calling the system time Warning

Calling the system time may be a possible source of non-determinism
docs/migration/seidb_archive_migration.md Outdated Show resolved Hide resolved
docs/migration/seidb_archive_migration.md Outdated Show resolved Hide resolved
docs/migration/seidb_archive_migration.md Outdated Show resolved Hide resolved
docs/migration/seidb_archive_migration.md Outdated Show resolved Hide resolved
docs/migration/seidb_archive_migration.md Outdated Show resolved Hide resolved
docs/migration/seidb_archive_migration.md Show resolved Hide resolved
docs/migration/seidb_archive_migration.md Outdated Show resolved Hide resolved
docs/migration/seidb_archive_migration.md Show resolved Hide resolved
fmt.Println("Scanning database and exporting leaf nodes...")
fmt.Println("SeiDB Archive Migration: Scanning database and exporting leaf nodes...")

startTimeTotal := time.Now() // Start measuring total time

Check warning

Code scanning / CodeQL

Calling the system time Warning

Calling the system time may be a possible source of non-determinism
@Kbhat1 Kbhat1 enabled auto-merge (squash) October 29, 2024 18:47
@Kbhat1 Kbhat1 disabled auto-merge October 29, 2024 19:12
@Kbhat1 Kbhat1 enabled auto-merge (squash) October 30, 2024 16:56
@Kbhat1 Kbhat1 merged commit dd940e5 into main Oct 30, 2024
47 of 49 checks passed
@Kbhat1 Kbhat1 deleted the ArchiveNodeMigrationOnline branch October 30, 2024 18:13
philipsu522 pushed a commit that referenced this pull request Oct 30, 2024
* Archive Node Online Migration

* Add migrate-iavl flag for online migration

* Update sei cosmos

* Add QMS for online migration (#1870)

* Add QMS for online migration

* Fix lint

---------

Co-authored-by: kbhat1 <[email protected]>

* Add migrate-height flag to start cmd

* Bump seidb

* Reduce logging

* Archive Migration doc

* Update form

* Update

* Export metric

* latest height

* Update start module logic

* Remove migrate SS command

* Enable ss store

* Update archive node migration docs

* More migration doc updates

* Update metrics in readme

* Merge main

* Remove dex

* Add Faq section

* Add requirements section

* Systemd instructions

* Minor add to readme

* Add more examples for migration height

* Bump sei cosmos

* Update error log

* Update command sc migration background

---------

Co-authored-by: Yiming Zang <[email protected]>
Kbhat1 added a commit that referenced this pull request Nov 1, 2024
* Archive Node Online Migration

* Add migrate-iavl flag for online migration

* Update sei cosmos

* Add QMS for online migration (#1870)

* Add QMS for online migration

* Fix lint

---------

Co-authored-by: kbhat1 <[email protected]>

* Add migrate-height flag to start cmd

* Bump seidb

* Reduce logging

* Archive Migration doc

* Update form

* Update

* Export metric

* latest height

* Update start module logic

* Remove migrate SS command

* Enable ss store

* Update archive node migration docs

* More migration doc updates

* Update metrics in readme

* Merge main

* Remove dex

* Add Faq section

* Add requirements section

* Systemd instructions

* Minor add to readme

* Add more examples for migration height

* Bump sei cosmos

* Update error log

* Update command sc migration background

---------

Co-authored-by: Yiming Zang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants