BlockImporter Error: worker died, SQLITE_CORRUPT

[ThomasBlock]

Hi. after a reboot my gateway seems to be in a broken state. i see this messages for block 1246846 in an endless loop.

can i somehow debug or should i delete all docker volumes?

core_1        | error: Error in StandaloneSqlite worker: database disk image is malformed {"code":"SQLITE_CORRUPT","stack":"SqliteError: database disk image is malformed\n    at StandaloneSqliteDatabaseWorker.<anonymous> (file:///app/dist/database/standalone-sqlite.js:419:60)\n    at StandaloneSqliteDatabaseWorker.sqliteTransaction [as insertBlockAndTxsFn] (/app/node_modules/better-sqlite3/lib/methods/transaction.js:65:24)\n    at StandaloneSqliteDatabaseWorker.saveBlockAndTxs (file:///app/dist/database/standalone-sqlite.js:623:14)\n    at MessagePort.<anonymous> (file:///app/dist/database/standalone-sqlite.js:1935:28)\n    at [nodejs.internal.kHybridDispatch] (node:internal/event_target:820:20)\n    at MessagePort.<anonymous> (node:internal/per_context/messageport:23:28)","timestamp":"2024-09-07T08:15:43.935Z"}
core_1        | error: Error importing block at height 1246846 Worker error {"class":"BlockImporter","stack":"Error: Worker error\n    at Worker.<anonymous> (file:///app/dist/database/standalone-sqlite.js:1619:32)\n    at Worker.emit (node:events:519:28)\n    at MessagePort.<anonymous> (node:internal/worker:263:53)\n    at [nodejs.internal.kHybridDispatch] (node:internal/event_target:820:20)\n    at MessagePort.<anonymous> (node:internal/per_context/messageport:23:28)","timestamp":"2024-09-07T08:15:43.935Z"}
core_1        | error: Uncaught exception: Worker error {"stack":"Error: Worker error\n    at Worker.<anonymous> (file:///app/dist/database/standalone-sqlite.js:1619:32)\n    at Worker.emit (node:events:519:28)\n    at MessagePort.<anonymous> (node:internal/worker:263:53)\n    at [nodejs.internal.kHybridDispatch] (node:internal/event_target:820:20)\n    at MessagePort.<anonymous> (node:internal/per_context/messageport:23:28)","timestamp":"2024-09-07T08:15:43.936Z"}
core_1        | info: Importing block... {"class":"BlockImporter","height":1246846,"timestamp":"2024-09-07T08:15:43.986Z"}
core_1        | error: Too many errors in StandaloneSqlite worker, exiting. {"timestamp":"2024-09-07T08:15:44.071Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-09-07T08:15:44.074Z"}
core_1        | error: Error importing block at height 1246846 worker died {"class":"BlockImporter","stack":"Error: worker died\n    at Worker.<anonymous> (file:///app/dist/database/standalone-sqlite.js:1634:41)\n    at Worker.emit (node:events:519:28)\n    at [kOnExit] (node:internal/worker:315:10)\n    at Worker.<computed>.onexit (node:internal/worker:229:20)","timestamp":"2024-09-07T08:15:44.074Z"}
core_1        | error: Uncaught exception: worker died {"stack":"Error: worker died\n    at Worker.<anonymous> (file:///app/dist/database/standalone-sqlite.js:1634:41)\n    at Worker.emit (node:events:519:28)\n    at [kOnExit] (node:internal/worker:315:10)\n    at Worker.<computed>.onexit (node:internal/worker:229:20)","timestamp":"2024-09-07T08:15:44.074Z"}

[ThomasBlock]

one way to solve:

rm data/sqlite/*
info: Importing block... {"class":"BlockImporter","height":214,"timestamp":"2024-09-07T22:41:05.990Z"}

lets see how long it takes..

[ThomasBlock]

After some months without problems, my node got again randomly corrupted ..

core_1        | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.463Z"}
core_1        | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.472Z"}
core_1        | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.476Z"}
core_1        | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.502Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.505Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.517Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.560Z"}
core_1        | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.564Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.586Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.616Z"}
core_1        | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.646Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.757Z"}
core_1        | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:55.353Z"}
core_1        | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:55.355Z"}
core_1        | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:55.356Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:55.383Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:55.385Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:55.393Z"}
core_1        | error: Error retrying missing transactions: {"class":"TransactionRepairWorker","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:55.394Z"}
core_1        | error: Uncaught exception: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "#<Object>". {"code":"ERR_UNHANDLED_REJECTION","name":"UnhandledPromiseRejection","stack":"UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason \"#<Object>\".\n    at throwUnhandledRejectionsMode (node:internal/process/promises:389:7)\n    at processPromiseRejections (node:internal/process/promises:470:17)\n    at process.processTicksAndRejections (node:internal/process/task_queues:96:32)","timestamp":"2024-11-27T07:06:55.394Z"}

this seems to help:

docker-compose down -v
docker system prune
rm -rf data/sqlite
docker-compose up -d

[dtfiedler]

Hi @ThomasBlock - before you remove the sqlite data, can you run a df -h on your server to see how much of the available file system is being used by your ar-ionode. It seems likely you are running of storage space causing SQLITE to fail writing new data.

[jim-toth]

After some months without problems, my node got again randomly corrupted ..

core_1        | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.463Z"}
core_1        | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.472Z"}
core_1        | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.476Z"}
core_1        | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.502Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.505Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.517Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.560Z"}
core_1        | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.564Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.586Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.616Z"}
core_1        | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:54.646Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:54.757Z"}
core_1        | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:55.353Z"}
core_1        | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:55.355Z"}
core_1        | error: Worker error {"class":"StandaloneSqliteDatabase","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:55.356Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:55.383Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:55.385Z"}
core_1        | error: Worker stopped with exit code 1 {"class":"StandaloneSqliteDatabase","exitCode":1,"timestamp":"2024-11-27T07:06:55.393Z"}
core_1        | error: Error retrying missing transactions: {"class":"TransactionRepairWorker","code":"SQLITE_ERROR","timestamp":"2024-11-27T07:06:55.394Z"}
core_1        | error: Uncaught exception: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "#<Object>". {"code":"ERR_UNHANDLED_REJECTION","name":"UnhandledPromiseRejection","stack":"UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason \"#<Object>\".\n    at throwUnhandledRejectionsMode (node:internal/process/promises:389:7)\n    at processPromiseRejections (node:internal/process/promises:470:17)\n    at process.processTicksAndRejections (node:internal/process/task_queues:96:32)","timestamp":"2024-11-27T07:06:55.394Z"}

this seems to help:

docker-compose down -v
docker system prune
rm -rf data/sqlite
docker-compose up -d

Try to see if all the db migrations ran correctly - that was my issue seeing these errors

[dtfiedler]

@jim-toth - can you share the command you ran to run the migration via docker?

[jim-toth]

@jim-toth - can you share the command you ran to run the migration via docker?

ye
docker-compose exec core /nodejs/bin/node dist/migrate.js up
you may need to prepend sudo or use docker compose depending on your docker setup

tho, all i used this for was to isolate the migration error i was getting. after fixing the issue by manually dropping the related table in the migration that didn't finish i just started the stack and let core migrate. the advantage of letting core migrate is that when it's done the node will just be up - some of these migrations take a long time ;)