Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable automatic backfilling of sqlite databases (msgindex, events, txhash) #11007

Closed
fridrik01 opened this issue Jun 26, 2023 · 2 comments
Closed

Comments

@fridrik01
Copy link
Contributor

Due to the importance of having these indices up-to-date from a performance perspective (API providers), we should make sure that we always have these databases fully backfilled.

Currently we don't automatically backfill the msgindex/events/txhas sqlite databases but instead rely on the user knowing to backfill manually. Having these databases fully backfilled is important to the performance of several RPC methods which otherwise have a huge latency (over 10+ minutes on archival nodes).

Instead, on Lotus startup, we should just always check if we have missing data (handling also cases where config was changed from disabled to enabled for an index) and then handle the backfilling in the background automatically. Only in some rare edge cases should user need to manually run backfill via lotus-shed, for example if they are migrating dbs and/or editing the raw sqlite data.

@rvagg
Copy link
Member

rvagg commented Jul 23, 2024

This probably needs something like MaxBackfillLookback for the case of an archival node turning this on late in the game; maybe you don't want to go back to FEVM and reprocess all tipsets to get events that you don't have.

@BigLep
Copy link
Member

BigLep commented Oct 4, 2024

This is being subsumed by work in #12453

@BigLep BigLep closed this as completed Oct 4, 2024
@github-project-automation github-project-automation bot moved this from ⌨️ In Progress to 🎉 Done in FilOz Oct 4, 2024
@github-project-automation github-project-automation bot moved this from In review to Done in PLDG Cohort 0 Project Board Oct 4, 2024
@rjan90 rjan90 moved this from 🎉 Done to ☑️ Done (Archive) in FilOz Oct 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: ☑️ Done (Archive)
Development

No branches or pull requests

6 participants