Indexer program stopped at reorganized block

[fewensa]

At present, it is a guess that it may be related to sql db, and more tests and information are needed to verify it.
The affected services are ponder, thegraph

URL: https://rpc.darwinia.network
Request body: {"method":"eth_getLogs","params":[{"blockHash":"0x49aae73c500163ba3ffd3a83596007baa6e75326a8ab847dec4726da781658dd"}]}

Details: unknown block
Version: [email protected]
    at withRetry.delay.count.count (file:///app/node_modules/viem/utils/buildRequest.ts:152:21)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at attemptRetry (file:///app/node_modules/viem/utils/promise/withRetry.ts:37:22)
caused by: RpcRequestError: RPC Request failed.

URL: https://rpc.darwinia.network
Request body: {"method":"eth_getLogs","params":[{"blockHash":"0x49aae73c500163ba3ffd3a83596007baa6e75326a8ab847dec4726da781658dd"}]}

Details: unknown block
Version: [email protected]
    at request (file:///app/node_modules/viem/clients/transports/http.ts:111:19)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at withRetry.delay.count.count (file:///app/node_modules/viem/utils/buildRequest.ts:129:18)
    at attemptRetry (file:///app/node_modules/viem/utils/promise/withRetry.ts:37:22)
caused by:
8:46:57 AM WARN  realtime   Failed to process 'darwinia' block 4747006

caused by: RpcRequestError: RPC Request failed.

URL: https://rpc.darwinia.network
Request body: {"method":"eth_getLogs","params":[{"blockHash":"0x49aae73c500163ba3ffd3a83596007baa6e75326a8ab847dec4726da781658dd"}]}

Details: unknown block
Version: [email protected]
    at request (file:///app/node_modules/viem/clients/transports/http.ts:111:19)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at withRetry.delay.count.count (file:///app/node_modules/viem/utils/buildRequest.ts:129:18)
    at attemptRetry (file:///app/node_modules/viem/utils/promise/withRetry.ts:37:22)
caused by:
8:46:57 AM WARN  realtime   Retrying 'darwinia' sync after 1 second.
8:48:26 AM WARN  realtime   Detected forked 'darwinia' block at height 4747035
8:48:27 AM WARN  realtime   Reconciled 2-block reorg on 'darwinia' with common ancestor block 4747033
8:49:31 AM WARN  realtime   Detected forked 'darwinia' block at height 4747046

[fewensa]

use thegraph
Crab index stopped, use sql backend

crab rpc endpoint: g1.crab2

{
  "data": {
    "_meta": {
      "hasIndexingErrors": false,
      "deployment": "QmZrSy6iUhj8a3cgZe4EQh2Ffncq3L2h1sKZD81RU7TQKR",
      "block": {
        "timestamp": null,
        "parentHash": null,
        "number": 4748739,
        "hash": "0x147f916d5499fa10a9c3fa91ea21e5f457a796b57ca266b4bf5f7eaca2f1df73"
      }
    }
  }
}

https://crab-scan.darwinia.network/block/0x147f916d5499fa10a9c3fa91ea21e5f457a796b57ca266b4bf5f7eaca2f1df73

thegraph rolling logs

Nov 29 03:05:27.025 INFO Committed write batch, time_ms: 2, weight: 344, entities: 0, block_count: 1, block_number: 4786279, sgd: 1, subgraph_id: QmaDnb8EmLUq4o5ZqNinkctEuysTPzrVtr3nKpCNTJJfW4, component: SubgraphInstanceManager
Nov 29 03:05:32.623 INFO Syncing 1 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 1, blocks_behind: 1, latest_block_head: 4786280, current_block_head: 4786279, provider: darwinia-rpc-0, component: EthereumPollingBlockIngestor
Nov 29 03:05:32.849 INFO Cleaned 1 blocks from the block cache. Only blocks with number greater than 4786270 remain, component: EthereumPollingBlockIngestor
Nov 29 03:05:33.355 INFO Committed write batch, time_ms: 2, weight: 344, entities: 0, block_count: 1, block_number: 4786280, sgd: 1, subgraph_id: QmaDnb8EmLUq4o5ZqNinkctEuysTPzrVtr3nKpCNTJJfW4, component: SubgraphInstanceManager
Nov 29 03:05:37.315 INFO Syncing 1 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 1, blocks_behind: 1, latest_block_head: 4749780, current_block_head: 4749779, provider: crab-rpc-1, component: EthereumPollingBlockIngestor
Nov 29 03:05:37.743 INFO Syncing 1 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 1, blocks_behind: 1, latest_block_head: 4786281, current_block_head: 4786280, provider: darwinia-rpc-0, component: EthereumPollingBlockIngestor
Nov 29 03:05:37.982 INFO Cleaned 1 blocks from the block cache. Only blocks with number greater than 4786271 remain, component: EthereumPollingBlockIngestor
Nov 29 03:05:38.332 INFO Syncing 1 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 1, blocks_behind: 1, latest_block_head: 4749781, current_block_head: 4749780, provider: crab-rpc-1, component: EthereumPollingBlockIngestor
Nov 29 03:05:38.849 INFO Committed write batch, time_ms: 2, weight: 344, entities: 0, block_count: 1, block_number: 4786281, sgd: 1, subgraph_id: QmaDnb8EmLUq4o5ZqNinkctEuysTPzrVtr3nKpCNTJJfW4, component: SubgraphInstanceManager
Nov 29 03:05:48.995 INFO Syncing 1 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 1, blocks_behind: 1, latest_block_head: 4786282, current_block_head: 4786281, provider: darwinia-rpc-0, component: EthereumPollingBlockIngestor
Nov 29 03:05:49.221 INFO Cleaned 1 blocks from the block cache. Only blocks with number greater than 4786272 remain, component: EthereumPollingBlockIngestor
Nov 29 03:05:49.725 INFO Committed write batch, time_ms: 2, weight: 344, entities: 0, block_count: 1, block_number: 4786282, sgd: 1, subgraph_id: QmaDnb8EmLUq4o5ZqNinkctEuysTPzrVtr3nKpCNTJJfW4, component: SubgraphInstanceManager
Nov 29 03:05:50.442 INFO Syncing 1 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 1, blocks_behind: 1, latest_block_head: 4786283, current_block_head: 4786282, provider: darwinia-rpc-0, component: EthereumPollingBlockIngestor
Nov 29 03:05:50.667 INFO Cleaned 1 blocks from the block cache. Only blocks with number greater than 4786273 remain, component: EthereumPollingBlockIngestor
Nov 29 03:05:51.171 INFO Committed write batch, time_ms: 2, weight: 344, entities: 0, block_count: 1, block_number: 4786283, sgd: 1, subgraph_id: QmaDnb8EmLUq4o5ZqNinkctEuysTPzrVtr3nKpCNTJJfW4, component: SubgraphInstanceManager
Nov 29 03:05:51.173 INFO Start pruning historical entities, latest_block: 4786283, earliest_block: 4786263, history_blocks: 20, sgd: 1, subgraph_id: QmaDnb8EmLUq4o5ZqNinkctEuysTPzrVtr3nKpCNTJJfW4, component: SubgraphInstanceManager
Nov 29 03:05:51.214 INFO Finished pruning entities, analyze_time_s: 0, time_s: 0, rows_copied: 0, rows_deleted: 0, tables: ø, sgd: 1, subgraph_id: QmaDnb8EmLUq4o5ZqNinkctEuysTPzrVtr3nKpCNTJJfW4, component: SubgraphInstanceManager
Nov 29 03:05:55.550 INFO Syncing 1 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 1, blocks_behind: 1, latest_block_head: 4786284, current_block_head: 4786283, provider: darwinia-rpc-0, component: EthereumPollingBlockIngestor
Nov 29 03:05:55.781 INFO Cleaned 1 blocks from the block cache. Only blocks with number greater than 4786274 remain, component: EthereumPollingBlockIngestor
Nov 29 03:05:56.282 INFO Committed write batch, time_ms: 2, weight: 344, entities: 0, block_count: 1, block_number: 4786284, sgd: 1, subgraph_id: QmaDnb8EmLUq4o5ZqNinkctEuysTPzrVtr3nKpCNTJJfW4, component: SubgraphInstanceManager

[fewensa]

dip7 indexer darwinia use kv-db (thegrpah)

{
  "data": {
    "_meta": {
      "hasIndexingErrors": false,
      "deployment": "QmaDnb8EmLUq4o5ZqNinkctEuysTPzrVtr3nKpCNTJJfW4",
      "block": {
        "timestamp": null,
        "parentHash": null,
        "number": 4829052,
        "hash": "0x8712bd739a66bf3d5397f17ef05f61528790fb4525617a0ab1d77dec46d2b91b"
      }
    }
  }
}