wip: use IPC for various things#1
Open
Sjors wants to merge 21 commits into
Open
Conversation
Introduce an experimental backend that talks to a multiprocess
`bitcoin-node` over a Unix socket using libmultiprocess Cap'n Proto
framing, as exposed by Bitcoin Core PR #29409 (interfaces::Chain).
Adds:
- a new optional `daemon_ipc_socket` config / CLI option
- a new `src/ipc.rs` module with an `IpcChain` handle that owns a
dedicated worker thread running a current-thread tokio runtime +
`LocalSet` (the generated capnp clients are not `Send`)
- capnp / capnp-rpc / tokio / tokio-util / futures dependencies plus
a path dep on the local `bitcoin-capnp-types` crate
When `daemon_ipc_socket` is set, `Daemon::connect` boots the
`IpcChain` worker and `Daemon::for_blocks` fetches each block via
`Chain.findBlock(wantData=true)` instead of using the P2P getblock
loop. The serialized form returned by libmultiprocess is the same
wire encoding electrs already consumes, so the downstream parsing
and indexing pipeline is unchanged.
When the option is absent (the default), the P2P path is preserved
verbatim, so this is a strict superset of existing behaviour.
Mirrors `tests/run.sh` but launches the multiprocess `bitcoin node` binary with `-ipcbind=unix` and points electrs at the resulting `<datadir>/regtest/node.sock` via `--daemon-ipc-socket`. The path to the multiprocess binary is configurable via `$BITCOIN_MULTIPROCESS_BIN`; if unset, the script looks for `bitcoin` on `PATH`. There is no hardcoded fallback.
- New `integration-ipc` job clones bitcoin/bitcoin#29409, builds the multiprocess `bitcoin` multi-call binary with capnproto enabled, installs Electrum + electrs, and invokes `tests/run-ipc.sh`. - The existing `build` and `integration` jobs now also clone the `bitcoin-capnp-types` chain-interface branch into a sibling directory to satisfy the `path = "../capnp-types"` Cargo dep introduced in this branch. Mirrored in both Dockerfiles.
Replaces a JSON-RPC `getblockchaininfo` call with the equivalent `Chain.havePruned` capnp method whenever the IPC backend is configured. The RPC fallback is retained for the non-IPC path so this remains a strict no-op for current deployments. This is the first migration of an existing electrs RPC call onto the PR #29409 chain interface; it is small but exercises the pattern end-to-end (config-conditional dispatch + integration test coverage).
When the IPC chain interface is available, fetch the block bytes via `Chain.findBlock` (already used for block download in for_blocks) and extract txids from the decoded block instead of issuing a JSON-RPC `getblock` call. This mirrors the call pattern of for_blocks and removes one more dependency on the JSON-RPC backend. The non-IPC path keeps the original `get_block_info` RPC call so this is a no-op for current deployments. Verified with both tests/run.sh (=== PASSED ===) and tests/run-ipc.sh (=== PASSED (IPC) ===).
…C is set Adds an `IpcChain::broadcast_transaction` wrapper around the `Chain.broadcastTransaction` capnp method and uses it from `Daemon::broadcast` when the IPC backend is configured. The method matches `sendrawtransaction`'s defaults: `MEMPOOL_AND_BROADCAST_TO_ALL` (broadcastMethod = 0) and `DEFAULT_MAX_RAW_TX_FEE = 0.10 BTC`. On failure the error string from the node is propagated. The non-IPC path keeps the original `send_raw_transaction` RPC call, so this is a no-op for current deployments. Verified end-to-end: the `payto & broadcast` step of tests/run-ipc.sh exercises the new path and reports === PASSED (IPC) ===.
Add IpcChain::find_locator_fork (encodes a CBlockLocator on the wire) and IpcChain::get_new_headers, which walks forward from the fork point using getBlockHash + findBlock(wantData=true) and parses the first 80 bytes of each block as a header. Cap the response at 2000 headers per call to mirror the P2P getheaders semantics. When --daemon-ipc-socket is set, Daemon::get_new_headers now goes through this IPC path instead of the P2P getheaders exchange in src/p2p.rs. Behaviour without IPC is unchanged. The chain interface does not expose a header-only fetch, so this downloads each full block twice (once for the header, again during the subsequent for_blocks indexing pass). Acceptable for an experimental backend over a local unix socket.
…IfTipChanged Restructure the IPC worker to spawn each Job onto the LocalSet so a long-running notifier task can run concurrently with regular RPC calls. Ctx now derives Clone (its two capnp clients are Rc-backed and cheap to clone) and is passed by value into each Job. Add IpcChain::start_block_notifier, which kicks off a worker-side loop that snapshots the current tip via getHeight + getBlockHash, blocks on Chain.waitForNotificationsIfTipChanged, and signals on a bounded(1) crossbeam channel each time the tip changes. Daemon::connect now starts this notifier when an IPC socket is configured and stashes the receiver on the Daemon. Daemon::new_block_notification returns the IPC receiver instead of the P2P new-block channel when IPC is available, eliminating the remaining P2P consumer for new-block notifications. Behaviour without IPC is unchanged.
Two unrelated cleanup-noise fixes in the integration scripts: - tests/run.sh: `bitcoin-cli stop` was sometimes racing the node's own exit and leaving a 'Could not connect to the server' line on stdout. Make the stop best-effort and fall back to killing the PID. - tests/run-ipc.sh: when the multiprocess node has IPC clients still attached at shutdown, the RPC `stop` request hangs indefinitely waiting for them to disconnect. The IPC client lives in electrs, which has already stopped above but whose capnp connection drop is not flushed back to the node. Skip the polite stop and just kill the node — this is a regtest fixture.
Now that header sync, block fetch, and new-block notifications all go through the Chain interface when --daemon-ipc-socket is set, there is no consumer left for the P2P connection. Make Daemon.p2p an Option and skip Connection::connect (and the network_active enforcement) on the IPC path. The non-IPC path is unchanged.
The Chain interface exposes no header-only fetch, so get_new_headers has to download each full block via findBlock(wantData=true) just to keep the first 80 bytes. Without a cache, the for_blocks pass that immediately follows refetches every one of those blocks over IPC -- doubling the byte count of initial sync, and making it a worse choice than P2P for any chain longer than a handful of blocks. Add a small bounded cache (256 MiB worth of payload, best-effort inserts on overflow) on IpcChain. get_new_headers stashes each block it fetches; get_block consults the cache and removes on hit, so for_blocks drains it in iteration order. Cache size is naturally bounded by the in-flight headers (capped at 100 per get_new_headers call -- much smaller than the P2P getheaders 2000, since each "header" here costs a full block, and mainnet blocks of up to ~4 MiB would otherwise blow the cache budget on a single batch) and falls back gracefully to direct fetch on overflow. A header-only chain method on the Bitcoin Core side would obviate this entirely; noted as a follow-up in EXPERIMENT.md.
Drop the `branches: [master]` filter on the `push` trigger so CI runs on every push, not just pushes to master. This makes the workflow fire for branch pushes on forks (e.g. Sjors/electrs experimental branches), matching the way most contributors iterate before opening a PR. PR triggers are unaffected.
…Fee} on IPC
Adds two thin wrappers on `IpcChain`:
- `estimate_smart_fee_sat_per_kvb(nblocks)` calls
`Chain.estimateSmartFee` (economical mode, no FeeCalculation), and
- `relay_min_fee_sat_per_kvb()` calls `Chain.relayMinFee`.
Both decode the serialized `CFeeRate` reply (FeeFrac { int64_t fee;
int32_t size; } -> 12 bytes LE, per Bitcoin Core SERIALIZE_METHODS)
into satoshis per kilo-vbyte via a new `decode_fee_rate_kvb` helper.
A `CFeeRate{}` (size == 0) reply maps to `Ok(None)` so the smart-fee
path can mirror the JSON-RPC -32603 "no estimate" behaviour.
`Daemon::estimate_fee` and `Daemon::get_relay_fee` now prefer the IPC
path when `ipc` is set and fall back to the bitcoind JSON-RPC client
otherwise.
This drops electrs's dependence on `getnetworkinfo.relayfee` and
`estimatesmartfee` for IPC deployments. Verified the `*_returns_default`
and `*_returns_decodable_blob` Chain integration tests in
bitcoin-capnp-types pass against the same multiprocess node electrs
talks to.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on:
Heavily vibe coded and it's possible that not all uses of IPC are actually beneficial.