Crawlers improvements. #924
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 8 commits
September 20, 2023 13:19
Added initial fix with event deduplication by selector instead of event hash. Functions call get selector from entry tags.
Refactor blocks cache.
1. Add new label_name_unverified which go to db without confirmatios. 2. Add insert as 1 operation in sql with deduplication. Blocks cache will include labels_table. As well state of database will go to blocks cahce once per from/to blocks range. Add delete from database label with label_name_unverified wich already exists ind database.
|
Depends on #919 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Immediate rows for Unverified Data
The Moonworm-crawler no longer waits for confirmations before insert to the database. Both events and transaction calls are saved instantly upon collecting the minimum number of blocks. Under label
{label_name}-unverified.Verification Process
During every crawl iteration, Moonworm-crawler runs a verification check for all events and transaction calls tagged with
unverified. It identifies these items and updates their labels after confirming their validity. This update:label_name_unverifiedto the designatedlabel_name.tx_hashandlog_index.label_name.Removal of Unverified Tags
Once verified, the crawler removes the
label_name_unverifiedwich already exists underlabel_name.Enhanced Blocks Cache for Events
The blocks cache now pulls its data from the database block and labels tables. It updates this data once for each block range.
Block Cache for Transaction Calls
No modifications were needed; however, future changes might require updates in the Moonworm library.
Event and Function Job Deduplication
Both events and functions have transitioned to using
abi_selectorinstead ofabi_hashto ensure uniqueness.