Internal/External
Internal if an IOG staff member.

Summary
With cardano-node 10.6.x and master, and nodes configured to use the V1LMDB backend, the nodes can get into a state where they stop to respond to queries made through cardano-cli. In such situation, queries like cardano-cli latest query tip hang indefinitely. While the queries stop working, the node doesn't produce any errors to log files, and blocks are still forged and adopted. When using strace on cardano-cli, it hangs on

futex(0x409f8cc, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY

Steps to reproduce
Steps to reproduce the behavior:

1. Follow https://github.com/IntersectMBO/cardano-node-tests?tab=readme-ov-file#%EF%B8%8F-running-tests-locally-with-nix
2. run the testsuite

NODE_REV="10.6.1" UTXO_BACKEND=disk KEEP_CLUSTERS_RUNNING=1 ./.github/regression.sh

The testsuite will hang if the issue manifests and is killed after 3 hours. The testnet instances keep running.
3. In another shell, identify the testnet that has the issue:

for i in {0..8}; do export CARDANO_NODE_SOCKET_PATH=$PWD/run_workdir/state-cluster$i/bft1.socket; timeout 5 cardano-cli latest query tip --testnet-magic 42 > /dev/null 2>&1; if [ "$?" -gt 120 ]; then { echo; echo; echo --- cluster $i ---; }; fi; done

There are 9 testnet instances in total.
4. Inspect the testnet. Files are in $PWD/run_workdir/state-cluster$i/*.{stdout,stderr}

For running the tests with custom cardano-node (or cardano-cli) binaries, create a .bin directory at the root of the tests repo and place them there.

System info (please complete the following information):

• OS Name: Fedora
• OS Version: 42
• Consensus version: the one in 10.6.1