use seed-chainstate.sh get error

[s4ndxyz]

I clone this repo and cp the .env then run sudo ./scripts/seed-chainstate.sh.
When it finished I run ./manage.sh -n mainnet -a start
But it shows errors:
stacks-blockchain-api | {"level":"error","message":"error processing core-node /new_block: Error: DB does not contain a parent block at height 102233 with index_hash 0xae4836064f818c24f325eac661c5c13f52aae5f8f5a0d97b980f6bb727bd3c0b DB does not contain a parent block at height 102233 with index_hash 0xae4836064f818c24f325eac661c5c13f52aae5f8f5a0d97b980f6bb727bd3c0b","stack":"Error: DB does not contain a parent block at height 102233 with index_hash 0xae4836064f818c24f325eac661c5c13f52aae5f8f5a0d97b980f6bb727bd3c0b\n at PgWriteStore.handleReorg (/app/src/datastore/pg-write-store.ts:2693:15)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at /app/src/datastore/pg-write-store.ts:215:7\n at scope (/app/node_modules/postgres/cjs/src/index.js:222:18)\n at Function.begin (/app/node_modules/postgres/cjs/src/index.js:209:14)\n at PgWriteStore.update (/app/src/datastore/pg-write-store.ts:213:5)\n at handleBlockMessage (/app/src/event-stream/event-server.ts:320:3)\n at run (/app/node_modules/p-queue/dist/index.js:163:29)","timestamp":"2023-04-18T03:45:45.659Z"} stacks-blockchain | ERRO [1681789545.660332] [testnet/stacks-node/src/event_dispatcher.rs:142] [chains-coordinator-0.0.0.0:20443] Event dispatcher: Failed POST, url: http://stacks-blockchain-api:3700/new_block, err: Response { status: InternalServerError, headers: {"keep-alive": "timeout=5", "date": "Tue, 18 Apr 2023 03:45:45 GMT", "content-type": "application/json; charset=utf-8", "x-powered-by": "Express", "etag": "W/\"c-3Rk1bge0s6VuJbi+S2m2iU0UGdY\"", "connection": "keep-alive", "content-length": "12"}, version: None, has_trailers: false, trailers_sender: Some(Sender { .. }), trailers_receiver: Some(Receiver { .. }), upgrade_sender: Some(Sender { .. }), upgrade_receiver: Some(Receiver { .. }), has_upgrade: false, body: Body { reader: "<hidden>", length: Some(12), bytes_read: 0 }, ext: Extensions, local_addr: None, peer_addr: None } stacks-blockchain-api | {"level":"error","message":"HTTP POST /new_block","req":{"headers":{"content-length":"109225","content-type":"application/json","host":"stacks-blockchain-api:3700"},"httpVersion":"1.1","method":"POST","originalUrl":"/new_block","query":{},"url":"/new_block"},"res":{"statusCode":500},"responseTime":9,"timestamp":"2023-04-18T03:45:45.660Z"}

[wileyj]

this error is specific to the API/postgres.
2 possibilities are top of mind:

1. there were errors running the import with seed-chainstate.sh, so the DB was not properly imported
2. the API had some issue writing or connecting to postgres after the node started.

I'm leaning towards option 1, but i'll need to see the (truncated, because it can be a lot) output from when that script was run. for 2, i've only seen that when used with seed-chainstate when there are reported errors restoring the DB data from the downloaded dump file.

[s4ndxyz]

I try it many times and it still shows the same error.
Here is my logs:

postgres.log
stacks-blockchain.log
stacks-blockchain-api.log

[wileyj]

I try it many times and it still shows the same error. Here is my logs:

postgres.log stacks-blockchain.log stacks-blockchain-api.log

right, but what i want to see is the output from seed-chainstate. my suspicion is that the DB data isn't being imported in full, or there are errors (which would cause the error you're seeing in the API)

from your API log:

{"level":"error","message":"error processing core-node /new_block: Error: DB does not contain a parent block at height 102233 with index_hash 0xae4836064f818c24f325eac661c5c13f52aae5f8f5a0d97b980f6bb727bd3c0b DB does not contain a parent block at height 102233 with index_hash 0xae4836064f818c24f325eac661c5c13f52aae5f8f5a0d97b980f6bb727bd3c0b","stack":"Error: DB does not contain a parent block at height 102233 with index_hash 0xae4836064f818c24f325eac661c5c13f52aae5f8f5a0d97b980f6bb727bd3c0b\n    at PgWriteStore.handleReorg (/app/src/datastore/pg-write-store.ts:2693:15)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at /app/src/datastore/pg-write-store.ts:215:7\n    at scope (/app/node_modules/postgres/cjs/src/index.js:222:18)\n    at Function.begin (/app/node_modules/postgres/cjs/src/index.js:209:14)\n    at PgWriteStore.update (/app/src/datastore/pg-write-store.ts:213:5)\n    at handleBlockMessage (/app/src/event-stream/event-server.ts:320:3)\n    at run (/app/node_modules/p-queue/dist/index.js:163:29)","timestamp":"2023-04-19T12:34:48.206Z"}

this tells me that the DB doesn't have block height 102233, which the blockchain has synced to - so the API will refuse to add more blocks to the DB until that row exists.
i need the output from seed-chainstate to see if there were errors importing the data

[s4ndxyz]

To save time in multiple attempts, I only commented out the delete file and download file operations
Here is my seed-chainstate.sh and seed-chainstate log.
seed-chainstate.sh.txt
seed-chainstate.log

[wileyj]

To save time in multiple attempts, I only commented out the delete file and download file operations Here is my seed-chainstate.sh and seed-chainstate log. seed-chainstate.sh.txt seed-chainstate.log

thanks! these both look fine, but the download urls are what i need (or the versions from your .env file).
for some reason, the chainstate data and the pg_dump data isn't aligned correctly and the only reason i've seen that happen is when using Hiro's archive and the latest file they provide (but, that's only based on timing of when they create the archive).

the two env vars i'm curious about:
STACKS_BLOCKCHAIN_API_VERSION
STACKS_BLOCKCHAIN_VERSION

[s4ndxyz]

I'm using the mainnet-blockchain-2.1.0.0.2-latest.tar.gz.sha256 content is:
c4b15816b48bdb2e6bd3e079da4af1b6c091bd4754bab28037b8f73fac5bc6bc /hirosystems/data/mainnet-stacks-blockchain-2.1.0.0.2-20230416.tar.gz
the stacks-blockchain-api-pg-15-7.1.2-latest.dump.sha256 content is:
0e714123cbe0512c7012dfeffcfcc813665f1a4c9319f827c6306110b488b8f0 /hirosystems/data/stacks-blockchain-api-pg-15-7.1.2-20230406.dump
I just run cp sample.env .env to make my .env
STACKS_BLOCKCHAIN_VERSION=2.1.0.0.2 STACKS_BLOCKCHAIN_API_VERSION=7.1.2

[wileyj]

got it, thanks! i'll see if i can reproduce. both of those versions should be fine (barring an issue with archive file itself).

[wileyj]

I'm using the mainnet-blockchain-2.1.0.0.2-latest.tar.gz.sha256 content is: c4b15816b48bdb2e6bd3e079da4af1b6c091bd4754bab28037b8f73fac5bc6bc /hirosystems/data/mainnet-stacks-blockchain-2.1.0.0.2-20230416.tar.gz the stacks-blockchain-api-pg-15-7.1.2-latest.dump.sha256 content is: 0e714123cbe0512c7012dfeffcfcc813665f1a4c9319f827c6306110b488b8f0 /hirosystems/data/stacks-blockchain-api-pg-15-7.1.2-20230406.dump I just run cp sample.env .env to make my .env STACKS_BLOCKCHAIN_VERSION=2.1.0.0.2 STACKS_BLOCKCHAIN_API_VERSION=7.1.2

i can reproduce this using the versions you've shared.
the issue that there is a mismatch in the archives Hiro is providing - it's something i've thought about how to handle in the script but haven't had time to try and find a solution yet.

essentially, the archive for the blockchain 2.1.0.0.2 was taking well after the last archive for api 7.1.2. as you can see below, the API is about 1k blocks behind the chainstate data (which produces the error you're seeing, since the blockchain is sending block 102351, and the API doesn't have 102350)

$ curl -sL localhost:3999/extended/v1/status  | jq -r .chain_tip.block_height
101037
$ curl -sL localhost:20443/v2/info  | jq -r .stacks_tip_height
102350

In testing, these versions were working fine - but there has been a hotfix and some other updates to the API.
in the script, i'm using the latest version of a tagged archive - and therein lies the problem. i think a better way would be to use the archive epoch, but taht would take some time to figure out (PR welcome though!).

in the meantime, i'd simply use the latest archives they have as your versions:

STACKS_BLOCKCHAIN_VERSION=2.1.0.0.3
STACKS_BLOCKCHAIN_API_VERSION=7.1.5

[wileyj]

i'll leave this open as a TODO, but i'm not sure when or even if it'll be possible to solve for this outside of always using teh latest versions of an archive Hiro provides

[pseudozach]

+1, just had the same issue.
any better way to run this script to avoid this issue?

[wileyj]

+1, just had the same issue. any better way to run this script to avoid this issue?

it'll take some thought, for now i've added in this line: https://github.com/stacks-network/stacks-blockchain-docker/blob/master/scripts/seed-chainstate.sh#L5

but it's not quite what i'm thinking is needed here, particularly when there is a new release but it hasn't been archived yet.
if you pull that release and try to use the script - it won't work because the archives don't exist yet.

moreover, using shell as I am here is very limiting. i may redo it soon in a language that would make switching to an older version/different epoch a bit easier.

that said - it does seem that the current (as of today) archives are broken (particualrly for testnet, i have not tested mainnet recently). no matter which version of the files i use, i'm stuck at testnet block 106195 on startup.
I don't think there is a solution that I provide here, but i'm speaking with Hiro about the above to see if they have an issue on their end that can be resolved.