Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multi chain rewind service #2673

Open
wants to merge 24 commits into
base: main
Choose a base branch
from
Open

multi chain rewind service #2673

wants to merge 24 commits into from
+1,234 −214

Conversation

yoozo
Copy link
Collaborator

@yoozo yoozo commented Feb 12, 2025 •
edited
Loading

Description

In a multi-chain project, after enabling historical mode, a rewind requires coordination with other processes for rollback. Below are the tasks we need to accomplish.

Fixes # (issue)
#2620

TODO

  • cold start requires rollback execution.
  • A pause mechanism is needed where all fetch processes stop in case of a rollback.
  • A logger notification is needed when the API fetch is paused.
  • Before performing the write, check if _global.rewind_lock exist. If they do, perform a rollback.
  • When starting the process, check if rewind_timestamp exists. If it does, perform a rewind.
  • Check if _global contain the rewind_timestamp. If none exist, release _global.rewind_lock.
  • Retrieve the corresponding block height to roll back to using a timestamp, It may require using the binary search method.
  • Use a DB notify/trigger to notify a cooperating process.
  • Extract global lock logic into a separate file.
  • Modify the lock handling logic to ensure correct release under concurrent conditions. (Add a rewindNum field to rewindLock to ensure the remaining processing count is correct when the lock is released.)
  • Test cases for related scenarios.

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Checklist

  • I have tested locally
  • I have performed a self review of my changes
  • Updated any relevant documentation
  • Linked to any relevant issues
  • I have added tests relevant to my changes
  • Any dependent changes have been merged and published in downstream modules
  • My code is up to date with the base branch
  • I have updated relevant changelogs. We suggest using chan

stwiname
stwiname reviewed Feb 17, 2025
View reviewed changes
Copy link
Collaborator

@stwiname stwiname left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. It would be good to have a test/simulation of the multichain rewind

@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 3, 2025 •
edited
Loading

Coverage report for .

Caution

Test run failed

St.❔
 Category Percentage Covered / Total
🟡 Statements
67.61% (-2.5% 🔻)
 21098/31205
🟡 Branches
78.83% (+0.53% 🔼)
 2833/3594
🟡 Functions
70.29% (+7.79% 🔼)
 1152/1639
🟡 Lines
67.61% (-2.5% 🔻)
 21098/31205
Show new covered files 🐣
St.❔
 File Statements Branches Functions Lines
🟢
... / migrate.ts
 85.34% 9.09% 100% 85.34%
🟢
... / migrate-abis.controller.ts
 95% 87.5% 100% 95%
🟢
... / constants.ts
 100% 100% 100% 100%
🟢
... / index.ts
 100% 100% 100% 100%
🟢
... / ethereum.ts
 82.28% 83.33% 100% 82.28%
🟢
... / migrate-manifest.controller.ts
 88% 72.41% 88.89% 88%
🟢
... / migrate-mapping.controller.ts
 100% 100% 100% 100%
🟢
... / migrate-controller.ts
 92.41% 85.19% 100% 92.41%
🟢
... / migrate.fixtures.ts
 100% 100% 100% 100%
🟢
... / migrate-schema.controller.ts
 97.23% 79.49% 100% 97.23%
🔴
... / types.ts
 0% 0% 0% 0%
🟢
... / createProject.fixtures.ts
 94.92% 77.78% 100% 94.92%
🟢
... / config.ts
 100% 100% 100% 100%
🟢
... / index.ts
 100% 100% 100% 100%
🔴
... / moduleLoader.ts
 56% 40% 100% 56%
🟢
... / types.ts
 100% 100% 100% 100%
🔴
... / utils.ts
 0% 0% 0% 0%
🟢
... / networkFamily.ts
 92.59% 88.89% 100% 92.59%
🟡
... / IPFSHTTPClientLite.ts
 76.47% 80% 88.89% 76.47%
🟢
... / index.ts
 100% 100% 100% 100%
🟢
... / utils.ts
 92% 66.67% 100% 92%
🟡
... / admin.controller.ts
 75.56% 96% 78.57% 75.56%
🟡
... / blockRange.ts
 74.6% 100% 80% 74.6%
🟢
... / index.ts
 100% 100% 100% 100%
🟢
... / blockchain.service.ts
 100% 100% 100% 100%
🟢
... / SubqueryProject.ts
 86.21% 83.33% 71.43% 86.21%
🟡
... / factory.ts
 79.22% 66.67% 100% 79.22%
🟢
... / core.module.ts
 100% 100% 100% 100%
🟢
... / dictionary.fixtures.ts
 100% 100% 100% 100%
🟢
... / GlobalData.entity.ts
 100% 100% 100% 100%
🟡
... / monitor.service.ts
 70.04% 87.21% 94.12% 70.04%
🟡
... / multiChainRewind.service.ts
 76.49% 77.42% 92.86% 76.49%
🔴
... / sandbox.service.ts
 33.33% 50% 25% 33.33%
🟡
... / baseCache.service.ts
 67.57% 90% 66.67% 67.57%
🟡
... / baseStoreModel.service.ts
 64.41% 81.82% 83.33% 64.41%
🟢
... / cacheable.ts
 89.29% 75% 100% 89.29%
🟡
... / csvStore.ts
 75% 73.68% 85.71% 75%
🟡
... / exporter.ts
 71.67% 71.43% 71.43% 71.67%
🟢
... / index.ts
 100% 100% 100% 100%
🟢
... / global.ts
 82.16% 76.92% 85.71% 82.16%
🔴
... / index.ts
 0% 0% 0% 0%
🟢
... / index.ts
 100% 100% 100% 100%
🟡
... / cacheMetadata.ts
 68.28% 81.82% 86.67% 68.28%
🟢
... / index.ts
 100% 100% 100% 100%
🟡
... / metadata.ts
 67.47% 77.78% 66.67% 67.47%
🟢
... / utils.ts
 100% 100% 100% 100%
🟢
... / cacheModel.ts
 88.86% 89.29% 90% 88.86%
🟢
... / index.ts
 100% 100% 100% 100%
🟢
... / model.ts
 81.01% 72% 90.91% 81.01%
🟢
... / utils.ts
 100% 100% 100% 100%
🟡
... / cachePoi.ts
 77.89% 89.47% 85.71% 77.89%
🟢
... / index.ts
 100% 100% 100% 100%
🔴
... / poi.ts
 41.96% 71.43% 44.44% 41.96%
🟢
... / setValueModel.ts
 83.97% 77.14% 85.71% 83.97%
🟡
... / storeCache.service.ts
 63.84% 65.52% 76.92% 63.84%
🟡
... / storeModel.service.ts
 64.84% 61.54% 71.43% 64.84%
🟢
... / types.ts
 100% 100% 100% 100%
🟢
... / utils.ts
 93.33% 60% 100% 93.33%
🟢
... / worker.core.module.ts
 100% 100% 0% 100%
🔴
... / worker.monitor.service.ts
 39.53% 100% 0% 39.53%
🔴
... / meta.controller.ts
 53.33% 100% 0% 53.33%
🟡
... / meta.module.ts
 60% 100% 0% 60%
🟡
... / process.ts
 61.29% 66.67% 40% 61.29%
🔴
... / foreceClean.init.ts
 47.62% 100% 0% 47.62%
🔴
... / reindex.init.ts
 33.33% 100% 0% 33.33%
🟢
... / testing.core.module.ts
 100% 100% 100% 100%
🟡
... / autoQueue.ts
 65.85% 86.49% 73.33% 65.85%
🟢
... / index.ts
 100% 100% 100% 100%
🔴
... / queue.ts
 47.22% 80% 66.67% 47.22%
🟡
... / rampQueue.ts
 78.72% 91.3% 87.5% 78.72%
🟢
... / string.ts
 100% 95.45% 100% 100%
🟡
... / blockchain.service.ts
 64.73% 100% 57.14% 64.73%
🔴
... / datasourceProcessors.ts
 0% 0% 0% 0%
🔴
... / index.ts
 0% 0% 0% 0%
🔴
... / types.ts
 0% 0% 0% 0%
🔴
... / networking.ts
 31.58% 100% 0% 31.58%
Show files with reduced coverage 🔻
St.❔
 File Statements Branches Functions Lines
🔴
... / delete.ts
48.72% (-5.57% 🔻)
 100% 0%
48.72% (-5.57% 🔻)
🔴
... / promote.ts
44.19% (-4.53% 🔻)
 100% 0%
44.19% (-4.53% 🔻)
🟡
... / add.ts
61.29% (-0.78% 🔻)
 100% 0%
61.29% (-0.78% 🔻)
🟢
... / publish.ts
83.72% (-5.61% 🔻)
 57.14% 100%
83.72% (-5.61% 🔻)
🔴
... / add-chain-controller.ts
39.85% (-0.23% 🔻)
 66.67% 30%
39.85% (-0.23% 🔻)
🟢
... / build-controller.ts
90.44% (-0.86% 🔻)
71.43% (+14.29% 🔼)
 100%
90.44% (-0.86% 🔻)
🟢
... / codegen-controller.ts
88.77% (-0.62% 🔻)
83.95% (+8.37% 🔼)
 100%
88.77% (-0.62% 🔻)
🟡
... / deploy-controller.ts
64.94% (+27.48% 🔼)
72.73% (-5.05% 🔻)
64.29% (+39.29% 🔼)
64.94% (+27.48% 🔼)
🟢
... / generate-controller.ts
97.75% (+0.1% 🔼)
93.24% (-0.79% 🔻)
 100%
97.75% (+0.1% 🔼)
🟡
... / init-controller.ts
73.06% (+2.57% 🔼)
54.55% (-13.45% 🔻)
61.11% (+7.78% 🔼)
73.06% (+2.57% 🔼)
🟢
... / project-controller.ts
83.13% (-14.13% 🔻)
42.86% (-17.14% 🔻)
75% (-25% 🔻)
83.13% (-14.13% 🔻)
🟢
... / publish-controller.ts
82.63% (+2.38% 🔼)
70.27% (-0.84% 🔻)
 100%
82.63% (+2.38% 🔼)
🟡
... / load.ts
79.41% (-8.82% 🔻)
66.67% (-19.05% 🔻)
 100%
79.41% (-8.82% 🔻)
🔴
... / models.ts
38.89% (-60.56% 🔻)
 92.31%
0% (-100% 🔻)
38.89% (-60.56% 🔻)
🔴
... / ProjectManifestVersioned.ts
52.05% (-32.88% 🔻)
 100%
53.85% (-4.49% 🔻)
52.05% (-32.88% 🔻)
🔴
... / model.ts
34.86% (-62.86% 🔻)
 100%
33.33% (-38.1% 🔻)
34.86% (-62.86% 🔻)
🔴
... / models.ts
52.94% (-47.06% 🔻)
 100%
0% (-100% 🔻)
52.94% (-47.06% 🔻)
🟡
... / load.ts
66.33% (+7.14% 🔼)
47.37% (-8.19% 🔻)
83.33% (+16.67% 🔼)
66.33% (+7.14% 🔼)
🟡
... / github-reader.ts
67.24% (-22.04% 🔻)
85.71% (+10.71% 🔼)
71.43% (-11.9% 🔻)
67.24% (-22.04% 🔻)
🟡
... / ipfs-reader.ts
63.49% (-22.22% 🔻)
72.73% (+9.09% 🔼)
71.43% (-11.9% 🔻)
63.49% (-22.22% 🔻)
🟡
... / local-reader.ts
72.09% (-13.95% 🔻)
100% (+37.5% 🔼)
83.33% (-16.67% 🔻)
72.09% (-13.95% 🔻)
🟢
... / reader.ts
81.58% (-7.89% 🔻)
57.14% (-17.86% 🔻)
 100%
81.58% (-7.89% 🔻)
🔴
... / base.ts
47.66% (-50.34% 🔻)
87.5% (-12.5% 🔻)
40% (-40% 🔻)
47.66% (-50.34% 🔻)
🔴
... / models.ts
31.74% (-68.26% 🔻)
 100%
0% (-100% 🔻)
31.74% (-68.26% 🔻)
🟢
... / NodeConfig.ts
88.05% (-0.56% 🔻)
78.13% (-1.5% 🔻)
83.33% (-2.71% 🔻)
88.05% (-0.56% 🔻)
🟢
... / ProjectUpgrade.service.ts
81.57% (-5.51% 🔻)
87.36% (-4% 🔻)
92.31% (+1% 🔼)
81.57% (-5.51% 🔻)
🟢
... / configure.module.ts
82.67% (-1.9% 🔻)
71.15% (+7.15% 🔼)
 100%
82.67% (-1.9% 🔻)
🟢
... / db.module.ts
89.71% (+1.42% 🔼)
62.5% (-20.83% 🔻)
 80%
89.71% (+1.42% 🔼)
🟢
... / SchemaMigration.service.ts
84.57% (-2.99% 🔻)
81.82% (+0.57% 🔼)
83.33% (-16.67% 🔻)
84.57% (-2.99% 🔻)
🟡
... / migration-helpers.ts
75.28% (-0.74% 🔻)
88.24% (+7.64% 🔼)
 90.91%
75.28% (-0.74% 🔻)
🟡
... / migration.ts
73.39% (-8.89% 🔻)
83.33% (+3.6% 🔼)
85% (-5.48% 🔻)
73.39% (-8.89% 🔻)
🟡
... / StoreOperations.ts
74.12% (-18.19% 🔻)
86.36% (+1.36% 🔼)
87.5% (-12.5% 🔻)
74.12% (-18.19% 🔻)
🔴
... / benchmark.service.ts
44.76% (-10.49% 🔻)
 66.67%
36.36% (-3.64% 🔻)
44.76% (-10.49% 🔻)
🔴
... / base-block-dispatcher.ts
18.3% (-29.05% 🔻)
 100%
4.55% (-5.45% 🔻)
18.3% (-29.05% 🔻)
🔴
... / block-dispatcher.ts
31.05% (-9.94% 🔻)
 66.67% 28.57%
31.05% (-9.94% 🔻)
🔴
... / worker-block-dispatcher.ts
18.47% (-31.74% 🔻)
 100%
0% (-27.27% 🔻)
18.47% (-31.74% 🔻)
🔴
... / coreDictionary.ts
48.72% (-51.28% 🔻)
87.5% (-6.94% 🔻)
77.78% (-22.22% 🔻)
48.72% (-51.28% 🔻)
🟡
... / dictionary.service.ts
67.49% (-17.66% 🔻)
80% (-5.71% 🔻)
83.33% (-7.58% 🔻)
67.49% (-17.66% 🔻)
🟡
... / dictionaryV1.ts
77.55% (-15.68% 🔻)
76.09% (-2.17% 🔻)
91.67% (-8.33% 🔻)
77.55% (-15.68% 🔻)
🟡
... / dictionaryV2.ts
70.62% (-11.06% 🔻)
68% (+2.78% 🔼)
88.89% (-11.11% 🔻)
70.62% (-11.06% 🔻)
🔴
... / ds-processor.service.ts
39.73% (-22.36% 🔻)
 75%
58.33% (-29.17% 🔻)
39.73% (-22.36% 🔻)
🟡
... / dynamic-ds.service.ts
63.38% (-26.82% 🔻)
63.64% (-3.03% 🔻)
81.82% (-3.9% 🔻)
63.38% (-26.82% 🔻)
🟢
... / fetch.service.ts
90.46% (-1.82% 🔻)
88.16% (-0.57% 🔻)
84.62% (-2.34% 🔻)
90.46% (-1.82% 🔻)
🟢
... / inMemoryCache.service.ts
86.21% (-13.79% 🔻)
 100%
75% (-25% 🔻)
86.21% (-13.79% 🔻)
🔴
... / indexer.manager.ts
17.73% (-15.48% 🔻)
 100%
10% (-2.5% 🔻)
17.73% (-15.48% 🔻)
🟢
... / PoiBlock.ts
84.68% (-13.71% 🔻)
 96.55%
75% (-25% 🔻)
84.68% (-13.71% 🔻)
🔴
... / poi.service.ts
55.7% (+31.78% 🔼)
57.69% (-42.31% 🔻)
80% (+51.43% 🔼)
55.7% (+31.78% 🔼)
🔴
... / poiSync.service.ts
55.88% (-31% 🔻)
77.61% (+1.05% 🔼)
88.89% (-3.42% 🔻)
55.88% (-31% 🔻)
🟡
... / project.service.ts
74.39% (-5.61% 🔻)
72.73% (+2.55% 🔼)
85.71% (+1.71% 🔼)
74.39% (-5.61% 🔻)
🔴
... / sandbox.ts
59.73% (-8.25% 🔻)
56.25% (-0.89% 🔻)
57.14% (-4.4% 🔻)
59.73% (-8.25% 🔻)
🔴
... / store.service.ts
59.62% (-6.42% 🔻)
70.59% (+2.17% 🔼)
82.86% (+25.71% 🔼)
59.62% (-6.42% 🔻)
🔴
... / entity.ts
53.57% (-7.97% 🔻)
 100%
50% (+50% 🔼)
53.57% (-7.97% 🔻)
🔴
... / store.ts
31.03% (-1.17% 🔻)
62.5% (-37.5% 🔻)
33.33% (+33.33% 🔼)
31.03% (-1.17% 🔻)
🟢
... / test.runner.ts
95.06% (-4.27% 🔻)
77.78% (+4.44% 🔼)
75% (-25% 🔻)
95.06% (-4.27% 🔻)
🔴
... / testing.service.ts
11.54% (-17.03% 🔻)
 100% 0%
11.54% (-17.03% 🔻)
🔴
... / unfinalizedBlocks.service.ts
58.28% (-25.71% 🔻)
74.51% (-22.21% 🔻)
91.3% (-0.36% 🔻)
58.28% (-25.71% 🔻)
🟡
... / worker.builder.ts
61.97% (-17.22% 🔻)
 65.38% 80%
61.97% (-17.22% 🔻)
🔴
... / worker.cache.service.ts
48.78% (-9.76% 🔻)
 100% 0%
48.78% (-9.76% 🔻)
🔴
... / worker.connectionPoolState.manager.ts
38.21% (-22.76% 🔻)
 100% 0%
38.21% (-22.76% 🔻)
🔴
... / worker.dynamic-ds.service.ts
44.44% (-17.78% 🔻)
 100% 0%
44.44% (-17.78% 🔻)
🔴
... / worker.service.ts
21.24% (-23.4% 🔻)
 100% 0%
21.24% (-23.4% 🔻)
🔴
... / worker.ts
41.75% (-0.71% 🔻)
 100% 0%
41.75% (-0.71% 🔻)
🔴
... / worker.unfinalizedBlocks.service.ts
26.79% (-73.21% 🔻)
 100%
0% (-100% 🔻)
26.79% (-73.21% 🔻)
🔴
... / logger.ts
49.3% (-15.18% 🔻)
 33.33%
40% (+20% 🔼)
49.3% (-15.18% 🔻)
🔴
... / event.listener.ts
16.16% (-64.65% 🔻)
 100%
0% (-25% 🔻)
16.16% (-64.65% 🔻)
🔴
... / health.controller.ts
37.93% (-17.24% 🔻)
 100%
0% (-50% 🔻)
37.93% (-17.24% 🔻)
🔴
... / health.service.ts
21.05% (-36.48% 🔻)
 100%
0% (-20% 🔻)
21.05% (-36.48% 🔻)
🔴
... / meta.service.ts
35.19% (-34.34% 🔻)
100% (+20% 🔼)
0% (-40% 🔻)
35.19% (-34.34% 🔻)
🔴
... / ready.controller.ts
34.78% (-21.74% 🔻)
 100%
0% (-50% 🔻)
34.78% (-21.74% 🔻)
🔴
... / ready.service.ts
37.5% (-54.17% 🔻)
 100%
0% (-66.67% 🔻)
37.5% (-54.17% 🔻)
🔴
... / forceClean.service.ts
21.05% (-5.26% 🔻)
 100% 0%
21.05% (-5.26% 🔻)
🔴
... / reindex.service.ts
19.59% (-13.74% 🔻)
 100% 0%
19.59% (-13.74% 🔻)
🟡
... / blocks.ts
71.21% (-28.79% 🔻)
91.67% (+2.78% 🔼)
75% (-25% 🔻)
71.21% (-28.79% 🔻)
🟡
... / project.ts
73.29% (-0.09% 🔻)
75% (+0.42% 🔼)
 77.78%
73.29% (-0.09% 🔻)
🟢
... / promise.ts
88.14% (-3.63% 🔻)
100% (+13.64% 🔼)
71.43% (-14.29% 🔻)
88.14% (-3.63% 🔻)
🟡
... / reindex.ts
75.76% (+41.72% 🔼)
10% (-90% 🔻)
100% (+100% 🔼)
75.76% (+41.72% 🔼)
🔴
... / yargs.ts
43.38% (-2.11% 🔻)
 100% 14.29%
43.38% (-2.11% 🔻)
🟢
... / api.service.ts
86.71% (+3.2% 🔼)
78.69% (+6.47% 🔼)
80.77% (-5.9% 🔻)
86.71% (+3.2% 🔼)
🟢
... / apiPromise.connection.ts
80% (+10.92% 🔼)
61.54% (-18.46% 🔻)
66.67% (+23.81% 🔼)
80% (+10.92% 🔼)
🔴
... / index.ts
0% (-100% 🔻)
0% (-100% 🔻)
0% (-100% 🔻)
0% (-100% 🔻)
🟢
... / substrateDictionary.service.ts
93.52% (+2.95% 🔼)
38.46% (-34.27% 🔻)
100% (+20% 🔼)
93.52% (+2.95% 🔼)
🔴
... / types.ts
0% (-100% 🔻)
0% (-100% 🔻)
0% (-100% 🔻)
0% (-100% 🔻)
🟢
... / substrateDictionaryV1.ts
85.43% (-0.05% 🔻)
78.26% (+8.49% 🔼)
84.62% (+1.28% 🔼)
85.43% (-0.05% 🔻)
🟡
... / substrateDictionaryV2.ts
70.79% (-4.72% 🔻)
50% (-25% 🔻)
 33.33%
70.79% (-4.72% 🔻)
🔴
... / indexer.manager.ts
47.96% (-9.58% 🔻)
 100%
16.67% (+7.58% 🔼)
47.96% (-9.58% 🔻)
🟢
... / runtimeService.ts
95.29% (+3.09% 🔼)
83.33% (-5.56% 🔻)
100% (+20% 🔼)
95.29% (+3.09% 🔼)
🟡
... / types.ts
70.45% (-14.84% 🔻)
 100% 0%
70.45% (-14.84% 🔻)
🟢
... / project.ts
88.73% (-0.16% 🔻)
 68.18% 87.5%
88.73% (-0.16% 🔻)
🟢
... / buffer.ts
95.12% (-4.88% 🔻)
95.83% (-4.17% 🔻)
90.48% (+6.27% 🔼)
95.12% (-4.88% 🔻)
🟢
... / builder.ts
93.02% (+0.25% 🔼)
82.14% (-0.62% 🔻)
 100%
93.02% (+0.25% 🔼)
🟡
... / logger.ts
70.35% (+0.75% 🔼)
51.85% (-3.15% 🔻)
78.57% (+3.57% 🔼)
70.35% (+0.75% 🔼)

Test suite run failed

Failed tests: 1/618. Failed suites: 1/108. 
  ● substrate utils › return defined if no timestamp set extrinsic

    expect(received).toBeDefined()

    Received: undefined

      113 |       block1.block.header.hash.toString(),
      114 |     );
    > 115 |     expect(timestamp).toBeDefined();
          |                       ^
      116 |     await api.disconnect();
      117 |   });
      118 | });

      at Object.<anonymous> (packages/node/src/utils/substrate.test.ts:115:23)

Report generated by 🧪jest coverage report action from f1757d1

stwiname
stwiname reviewed Mar 4, 2025
View reviewed changes
packages/node-core/src/indexer/entities/GlobalData.entity.ts
export type RewindLockInfo = {
/** Timestamp to rewind to. */
timestamp: number;
chainNum: number;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does chainNum mean here? Should it be chainId?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it means the number of other chains that need to wait for rollback. Maybe we need to rename it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe chainsCount?

yoozo and others added 14 commits March 5, 2025 09:30
@yoozo yoozo requested a review from stwiname March 9, 2025 13:38
stwiname
stwiname reviewed Mar 10, 2025
View reviewed changes
packages/node-core/src/db/db.module.ts
@@ -3,6 +3,7 @@

import {DynamicModule, Global} from '@nestjs/common';
import {Sequelize, Options as SequelizeOption} from '@subql/x-sequelize';
import {PoolConfig} from 'pg';
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be removed.

packages/node-core/src/db/sync-helper.test.ts
let sequelize: Sequelize;
const schema = 'multi-chain-test';

let client: unknown;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please provide the type for this.

packages/node-core/src/db/sync-helper.test.ts
Comment on lines +265 to +271
await delay(1);

await sequelize.query(rewindSqlFromTimestamp(rewindTimestamp - 1));
await delay(1);

await sequelize.query(`DELETE FROM "${schema}"."_global" WHERE "key" = '${RewindLockKey}'`);
await delay(1);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please try and remove the delays. Or reduce them to much smaller times

packages/node-core/src/indexer/blockDispatcher/base-block-dispatcher.ts
Comment on lines +229 to +232
const timestamp = await this.blockChainService.getBlockTimestamp(blockPayload.height);
this._pendingRewindHeader = {
blockHeight: Number(blockPayload.height),
timestamp,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think here now its easiest to call getHeaderForHeight

packages/node-core/src/indexer/entities/GlobalData.entity.ts
export type RewindLockInfo = {
/** Timestamp to rewind to. */
timestamp: number;
chainNum: number;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe chainsCount?

packages/node-core/src/indexer/store.service.ts
Comment on lines +512 to +514
if ((await this.getRewindTimestamp()) !== undefined) return;
const rewindTimestampKey = generateRewindTimestampKey(this.subqueryProject.network.chainId);
await this.globalDataRepo.create({key: rewindTimestampKey, value: 0});
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There could be an update between getting and setting meaning that values could be overwritten. It would be good to make this an atomic operation.

packages/node-core/src/indexer/storeModelProvider/global/global.ts
return JSON.stringify({timestamp: rewindTimestamp, chainNum: chainTotal});
}

async getGlobalRewindStatus() {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please specify the return type.

packages/node-core/src/indexer/worker/worker.service.ts
@@ -33,7 +33,7 @@ export abstract class BaseWorkerService<
private queue: AutoQueue<IBlock<B>>;

protected abstract fetchChainBlock(heights: number, extra: E): Promise<IBlock<B>>;
protected abstract toBlockResponse(block: B): R;
protected abstract toBlockResponse(block: B): Promise<R>;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the reason for this change?

packages/node/src/utils/substrate.ts
Comment on lines +446 to +460
function isFullHeader(header: OptionalTsHeader): header is Header {
return header.timestamp !== undefined;
}

export async function fillTsInHeader(
api: ApiPromise,
header: OptionalTsHeader,
): Promise<Header> {
if (!isFullHeader(header)) {
const timestamp = await getTimestampFromBlockHash(api, header.blockHash);
return { ...header, timestamp };
}
return header;
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this redundant now that timestamp is required?

packages/node/src/utils/substrate.ts
Comment on lines +103 to +106
const blockTimestamp = await (
await api.at(blockHash)
).query.timestamp.now();
return new Date(blockTimestamp.toNumber());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this needed? I thought for Shiden this still didn't work?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stwiname stwiname stwiname left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@yoozo @stwiname