cardano-testnet: fail fast with a diagnosis when the chain stalls irrecoverably

[palas]

Another attempt at reducing/avoiding the flakiness of tests by stopping the waits when the chain is dead.

Description

A cardano-testnet network permanently stops producing blocks whenever no block is forged for longer than the ledger view forecast horizon of 3 * securityParam / activeSlotsCoeff slots - 300 slots = 30s of wall clock with the default genesis options (securityParam=5, activeSlotsCoeff=0.05, slotLength=0.1s). Past that point every node fails its leadership check with Forge.Loop.NoLedgerView and the chain can never extend again. This happens when node startup exceeds the start-time offset (startTimeOffsetSeconds, 15s) plus the horizon, or when an overloaded machine starves the nodes of CPU for longer than the horizon mid-run, and it made tests hang forever in chain-progress waits (waitForEpochs/waitForBlocks/waitUntilEpoch) whose deadlines are measured in epochs/blocks/slots that never arrive.

Detect the situation instead, as early as it is provable, and explain it:

• chainForecastHorizon computes the horizon from the ShelleyGenesis backing the testnet (read via the node configuration); the stall threshold is max 60s (2 * horizon) - 60s with the default genesis - falling back to a conservative 300s when the genesis cannot be read.
• retryUntilRightM: when no chain-state update arrives for the whole stall threshold (previously a silently swallowed 300s fallback), fail with a message explaining the mechanism, the actual horizon, and pointing at the node logs.
• waitUntilEpoch: race foldEpochState against a stall watchdog with the same semantics and diagnostic.
• waitForBlockThrow: the startup deadline is now derived from the genesis (startTimeOffsetSeconds + horizon + 15s margin instead of a flat 45s) and the error explains the likely cause and remedy.

In addition to that:

Two teardown hardenings for overloaded machines, where a test suite run can wedge with no output at all - not even the per-test 600s watchdogs fire visibly - until the CI-level timeout kills it.

• initiateProcess now releases processes with cleanupProcessBounded instead of cleanupProcess. The bounded variant waits a 15s grace period and escalates to SIGKILL, which cannot be ignored or blocked, so node termination is guaranteed; if the process still survives (uninterruptible kernel sleep) it is leaked rather than waited on.

• asyncRegister_ no longer waits unboundedly for the cancelled thread: H.cancel blocks until the thread finishes, and resource release actions run with synchronous exceptions masked, so a thread stuck in a foreign call would wedge the teardown forever with nothing able to interrupt it. Cap the wait at 15s and leak the thread instead.

Attempts to fix #5762 (the hangs)

Checklist

• Commit sequence broadly makes sense and commits have useful messages
• New tests are added if needed and existing tests are updated. These may include:
  • golden tests
  • property tests
  • roundtrip tests
  • integration tests
    See Running tests for more details
• Any changes are noted in the CHANGELOG.md for affected package
• The version bounds in .cabal files are updated
• CI passes. See note on CI. The following CI checks are required:
  • Code is linted with hlint. See .github/workflows/check-hlint.yml to get the hlint version
  • Code is formatted with stylish-haskell. See .github/workflows/stylish-haskell.yml to get the stylish-haskell version
  • Code builds on Linux, MacOS and Windows for ghc-9.6 and ghc-9.12
• Self-reviewed the diff