Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sync Mutex Unlock Panic/Crash Loop #165

Open
crypto-services opened this issue Dec 27, 2024 · 0 comments
Open

Sync Mutex Unlock Panic/Crash Loop #165

crypto-services opened this issue Dec 27, 2024 · 0 comments

Comments

@crypto-services
Copy link

Version v1.13.3, Ubuntu 22.04.

When a syncing node reaches the head there is a condition where an unlocked mutex is trying to be unlocked resulting in a panic/crash loop.

Logs:

Fri 2024-12-27 05:37:39 UTC injectived[2057953]: 5:37AM INF Replay last block using real app module=consensus
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: 5:37AM INF finalized block block_app_hash=3AE4F30C728D0E8BAB9CB2B997C959C3E0BE4D2C6DD4E5D415C97C51B8A7DBDD height=99082225 module=consensus num_txs_res=153 num_val_updates=0
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: 5:37AM INF executed block app_hash=3AE4F30C728D0E8BAB9CB2B997C959C3E0BE4D2C6DD4E5D415C97C51B8A7DBDD height=99082225 module=consensus
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: fatal error: sync: unlock of unlocked mutex
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 1 [running]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: sync.fatal({0x5ae31ab?, 0x8?})
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /usr/local/go/src/runtime/panic.go:1007 +0x18
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: sync.(*Mutex).unlockSlow(0xc0021962a0, 0xffffffff)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /usr/local/go/src/sync/mutex.go:229 +0x35
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: sync.(*Mutex).Unlock(...)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /usr/local/go/src/sync/mutex.go:223
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cosmos/iavl.(*BatchWithFlusher).Set(0xc0021962a0, {0xc008f00810, 0xd, 0xd}, {0xc008fef100, 0xf2, 0x100})
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/cosmos/[email protected]/batch.go:62 +0x16d
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cosmos/iavl.(*nodeDB).SaveNode(0xc001ae4340, 0xc001a746e0)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:231 +0x2bd
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cosmos/iavl.(*MutableTree).saveNewNodes(0xc002da8190, 0x5a93a7d?)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:1056 +0x113
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cosmos/iavl.(*MutableTree).SaveVersion(0xc002da8190)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:764 +0x576
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: cosmossdk.io/store/iavl.(*Store).Commit(0xc00488bec0?)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/cosmos-sdk/[email protected]/iavl/store.go:126 +0x78
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: cosmossdk.io/store/rootmulti.commitStores(0x5e7dff1, 0xc001e2bd10, 0xc00339d6b0)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/cosmos-sdk/[email protected]/rootmulti/store.go:1205 +0x158
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: cosmossdk.io/store/rootmulti.(*Store).Commit(0xc0005ccfc8)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/cosmos-sdk/[email protected]/rootmulti/store.go:491 +0x1a5
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc0005cd208)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/[email protected]/baseapp/abci.go:953 +0x2e2
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cosmos/cosmos-sdk/server.cometABCIWrapper.Commit(...)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/[email protected]/server/cmt_abci.go:56
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cometbft/cometbft/abci/client.(*localClient).Commit(0xc001f411c0?, {0x6c5d330?, 0x95a7960?}, 0x7f1b442e6c28?)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/[email protected]/abci/client/local_client.go:113 +0xcd
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cometbft/cometbft/proxy.(*appConnConsensus).Commit(0xc003122e88, {0x6c5d330, 0x95a7960})
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/[email protected]/proxy/app_conn.go:109 +0x168
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cometbft/cometbft/state.(*BlockExecutor).Commit(_, {{{0xb, 0x0}, {0xc001e317f0, 0x7}}, {0xc001e31800, 0xb}, 0x1, 0x5e7dff1, {{0xc004a20ea0, ...}, ...}, ...}, ...)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/[email protected]/state/execution.go:415 +0x1a5
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cometbft/cometbft/state.(*BlockExecutor).applyBlock(_, {{{0xb, 0x0}, {0xc001e317f0, 0x7}}, {0xc001e31800, 0xb}, 0x1, 0x5e7dff1, {{0xc004a20ea0, ...}, ...}, ...}, ...)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/[email protected]/state/execution.go:298 +0xf10
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cometbft/cometbft/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc001e317f0, 0x7}}, {0xc001e31800, 0xb}, 0x1, 0x5e7dff0, {{0xc0024de3c0, ...}, ...}, ...}, ...)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/[email protected]/state/execution.go:227 +0x265
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cometbft/cometbft/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc001e317f0, 0x7}}, {0xc001e31800, 0xb}, 0x1, 0x5e7dff0, {{0xc0024de3c0, ...}, ...}, ...}, ...)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/[email protected]/consensus/replay.go:534 +0x24e
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cometbft/cometbft/consensus.(*Handshaker).ReplayBlocksWithContext(_, {_, _}, {{{0xb, 0x0}, {0xc001e317f0, 0x7}}, {0xc001e31800, 0xb}, 0x1, ...}, ...)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/[email protected]/consensus/replay.go:433 +0x77a
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cometbft/cometbft/consensus.(*Handshaker).HandshakeWithContext(0xc00c337120, {0x6c5d288, 0xc00181fc70}, {0x6c865d8, 0xc002d782a0})
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/[email protected]/consensus/replay.go:274 +0x425
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cometbft/cometbft/node.doHandshake({_, _}, {_, _}, {{{0xb, 0x0}, {0xc001e317f0, 0x7}}, {0xc001e31800, 0xb}, ...}, ...)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/[email protected]/node/setup.go:186 +0x1a9
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cometbft/cometbft/node.NewNodeWithContext({0x6c5d288, 0xc00181fc70}, 0xc002c56500, {0x6c2f5d0, 0xc000f3d900}, 0xc001ae86f0, {0x6c118c0, 0xc004f27230}, 0xc002530210, 0x63ddce0, ...)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/[email protected]/node/node.go:360 +0x630
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: main.startCmtNode({0x6c5d288, 0xc00181fc70}, 0xc002c56500, {0x6caa048, 0xc0025ae008}, 0xc001e0be80)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/actions-runner/_work/injective-core/injective-core/injective-core/cmd/injectived/start.go:304 +0x42f
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: main.startInProcess(_, {{{0xc00146de00, 0xc}, 0x0, {0xc0034e2330, 0x7}, {0x5b068d4, 0x1}, {0x5b068d4, 0x1}, ...}, ...}, ...)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/actions-runner/_work/injective-core/injective-core/injective-core/cmd/injectived/start.go:542 +0x1f9
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: main.start(_, {{0x0, 0x0, 0x0}, {0x6c8abf0, 0xc002e04990}, 0x0, {0xc002a31500, 0xb}, {0x6caa1d8, ...}, ...}, ...)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/actions-runner/_work/injective-core/injective-core/injective-core/cmd/injectived/start.go:212 +0x2fe
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: main.StartCmdWithOptions.func2.1()
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/actions-runner/_work/injective-core/injective-core/injective-core/cmd/injectived/start.go:170 +0x63
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: main.wrapCPUProfile(0xc001e0be80, 0xc0028eba10)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/actions-runner/_work/injective-core/injective-core/injective-core/cmd/injectived/start.go:768 +0x1b3
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: main.StartCmdWithOptions.func2(0xc0033c2608, {0xc001e8a1a0?, 0x0?, 0x2?})
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/actions-runner/_work/injective-core/injective-core/injective-core/cmd/injectived/start.go:169 +0x212
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/spf13/cobra.(*Command).execute(0xc0033c2608, {0xc001e8a180, 0x2, 0x2})
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0xaca
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/spf13/cobra.(*Command).ExecuteC(0xc0033b6008)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x3ff
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/spf13/cobra.(*Command).Execute(...)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/spf13/cobra.(*Command).ExecuteContext(...)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/spf13/[email protected]/command.go:1032
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: main.Execute(0xc0033b6008)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/actions-runner/_work/injective-core/injective-core/injective-core/cmd/injectived/root.go:176 +0x167
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: main.main()
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/actions-runner/_work/injective-core/injective-core/injective-core/cmd/injectived/main.go:10 +0x25
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 5 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/desertbit/timer.timerRoutine()
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/desertbit/[email protected]/timers.go:119 +0xac
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/desertbit/timer.init.0 in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/desertbit/[email protected]/timers.go:15 +0x1a
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 18 [syscall]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: os/signal.signal_recv()
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /usr/local/go/src/runtime/sigqueue.go:152 +0x29
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: os/signal.loop()
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /usr/local/go/src/os/signal/signal_unix.go:23 +0x13
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by os/signal.Notify.func1.1 in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /usr/local/go/src/os/signal/signal.go:151 +0x1f
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 19 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: go.opencensus.io/stats/view.(*worker).start(0xc000940e80)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/[email protected]/stats/view/worker.go:292 +0x9f
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by go.opencensus.io/stats/view.init.0 in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x8d
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 20 [sync.Cond.Wait]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: sync.runtime_notifyListWait(0xc00060b350, 0x0)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /usr/local/go/src/runtime/sema.go:569 +0x159
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: sync.(*Cond).Wait(0x0?)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /usr/local/go/src/sync/cond.go:70 +0x85
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cihub/seelog.(*asyncLoopLogger).processItem(0xc00057e630)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/cihub/[email protected]/behavior_asynclooplogger.go:50 +0x99
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cihub/seelog.(*asyncLoopLogger).processQueue(0xc00057e630)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/cihub/[email protected]/behavior_asynclooplogger.go:63 +0x33
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/cihub/seelog.NewAsyncLoopLogger in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/cihub/[email protected]/behavior_asynclooplogger.go:40 +0xcf
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 21 [sync.Cond.Wait]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: sync.runtime_notifyListWait(0xc00060b510, 0x0)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /usr/local/go/src/runtime/sema.go:569 +0x159
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: sync.(*Cond).Wait(0x0?)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /usr/local/go/src/sync/cond.go:70 +0x85
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cihub/seelog.(*asyncLoopLogger).processItem(0xc00057e750)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/cihub/[email protected]/behavior_asynclooplogger.go:50 +0x99
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cihub/seelog.(*asyncLoopLogger).processQueue(0xc00057e750)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/cihub/[email protected]/behavior_asynclooplogger.go:63 +0x33
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/cihub/seelog.NewAsyncLoopLogger in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/cihub/[email protected]/behavior_asynclooplogger.go:40 +0xcf
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 144 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc002d8aa50)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session_util.go:189 +0x59e
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.newSession in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:93 +0x296
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 145 [runnable]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc002da3500)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:124 +0x25b
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:148 +0x447
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 338 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc002da3500)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_state.go:101 +0x9c
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:149 +0x485
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 340 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc002da3500)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:773 +0x105
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:156 +0x536
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 341 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc002d8ab40)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session_util.go:189 +0x59e
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.newSession in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:93 +0x296
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 342 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc0033216c0)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:91 +0x149
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:148 +0x447
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 343 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc0033216c0)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_state.go:101 +0x9c
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:149 +0x485
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 344 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc0033216c0)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:836 +0x6ca
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:155 +0x4f6
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 345 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc0033216c0)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:773 +0x105
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:156 +0x536
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 385 [chan receive]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cosmos/cosmos-sdk/server.ListenForQuitSignals.func1()
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/[email protected]/server/util.go:423 +0x46
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cosmos/cosmos-sdk/server.ListenForQuitSignals.func2()
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/[email protected]/server/util.go:431 +0x13
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: golang.org/x/sync/errgroup.(*Group).Go.func1()
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 383 [sleep]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: time.Sleep(0x3b9aca00)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /usr/local/go/src/runtime/time.go:195 +0x115
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/hashicorp/go-metrics.(*Metrics).collectStats(0xc0035e4d20)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/hashicorp/[email protected]/metrics.go:270 +0x1f
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/hashicorp/go-metrics.New in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/hashicorp/[email protected]/start.go:84 +0x169
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 384 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/xlab/closer.(*closer).wait(0xc000222090)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/xlab/[email protected]/closer.go:104 +0xb0
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/xlab/closer.Init in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/xlab/[email protected]/closer.go:227 +0x169
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 418 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc0035e4e10)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session_util.go:189 +0x59e
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.newSession in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:93 +0x296
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 381 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/hashicorp/go-metrics.(*InmemSignal).run(...)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/hashicorp/[email protected]/inmem_signal.go:64
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/hashicorp/go-metrics.NewInmemSignal in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/hashicorp/[email protected]/inmem_signal.go:38 +0x145
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 419 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc004cc2fc0)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:91 +0x149
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:148 +0x447
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 420 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc004cc2fc0)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_state.go:101 +0x9c
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:149 +0x485
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 421 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc004cc2fc0)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:836 +0x6ca
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:155 +0x4f6
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 422 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc004cc2fc0)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:773 +0x105
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:156 +0x536
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 423 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc002b7c690)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session_util.go:189 +0x59e
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.newSession in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session.go:93 +0x296
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 424 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc004bd3180)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:91 +0x149
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:148 +0x447
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 425 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc004bd3180)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_state.go:101 +0x9c
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:149 +0x485
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 426 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc004bd3180)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:836 +0x6ca
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:155 +0x4f6
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 427 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc004bd3180)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:773 +0x105
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:156 +0x536
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 483 [select]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cometbft/cometbft/proxy.(*multiAppConn).killTMOnClientError(0xc002d782a0)
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/[email protected]/proxy/multi_app_conn.go:140 +0x110
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/cometbft/cometbft/proxy.(*multiAppConn).OnStart in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/[email protected]/proxy/multi_app_conn.go:120 +0x385
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: goroutine 484 [chan receive]:
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: github.com/cometbft/cometbft/libs/pubsub.(*Server).loop(0xc002a91a40, {0xc0023e95c0, 0xc0023e95f0})
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/[email protected]/libs/pubsub/pubsub.go:323 +0x88
Fri 2024-12-27 05:37:41 UTC injectived[2057953]: created by github.com/cometbft/cometbft/libs/pubsub.(*Server).OnStart in goroutine 1
Fri 2024-12-27 05:37:41 UTC injectived[2057953]:         /root/go/pkg/mod/github.com/!injective!labs/[email protected]/libs/pubsub/pubsub.go:309 +0x90
Fri 2024-12-27 05:37:41 UTC systemd[1]: injectived.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@crypto-services and others