Stratum v2 Template Provider via IPC interface (multiprocess)

[Sjors]

This is an alternative approach to #68. Rather than integrating the Template Provider directly into Bitcoin Core, we make it an entirely separate application that communicates via IPC, see design/multiprocess.md.

Usage

Compile as usual, multiprocess is enabled by default.

Alternatively get a recent binary, see (unofficial, by me) releases ending in "ICP".

Start node:

build/src/bitcoin-node -chain=testnet4 -ipcbind=unix

Start the Template Provider:

build/bin/bitcoin-mine -chain=testnet4 -debug=sv2 -loglevel=sv2:trace

See general Stratum v2 usage instructions in bitcoin#29432.

Rationale, PRs this builds on

Upstream PRs used:

• Add bitcoin-{node,gui} to release binaries for IPC bitcoin/bitcoin#31802
• ipc: add bitcoin-mine test program bitcoin/bitcoin#30437

There are multiple ways to create such an application. It could be built from scratch in Rust, using bits and pieces of the Stratum Reference Implementation SRI. It could be a c++ application built from scratch, perhaps using some libraries from Bitcoin Core.

But this PR goes about in a slightly different way. Similar to Elements or Knots it's a set of patches on top of the full Bitcoin Core codebase. This allows me to leverage the build system, subclass Transport and use various other useful bits.

This lets me reuse almost all code from the integrated Template Provider bitcoin#29432. Both build direct on:

• Sv2 noise #66
• Stratum v2 Transport #67
• Stratum v2 connman #50
• Stratum v2 Template Provider common functionality #49

I modified the guix script to only produce bitcoin-node (for now) and bitcoin-mine binaries.

Goal

The goal is to drop bitcoin-node from the release here. Ideally users would install Bitcoin Core in the manner they're familiar with. They would then install bitcoin-mine separately and it should just work(tm), at most having to add a line to bitcoin.conf to turn IPC on.

This won't work until we ship a release with multiprocess and the mining interface. See bitcoin#30437 for progress towards that.

Rebase hint

Rebase hint:

git reset --hard 2024/07/sv2-tp-common
git fetch sjors
git merge 2025/02/ipc-yea
git fetch ryanofsky
git merge ryanofsky/pr/mine

git cherry-pick sjors/sv2-ipc~12^..sjors/sv2-ipc

[Sjors]

@ryanofsky I slightly mangled e373642 because of bitcoin@4e1a434#r1681193702. I'll clean that up later.

[Sjors]

Incorporated the changes from bitcoin#30437 (comment) and cleaned up commit history.

[Sjors]

@ryanofsky do you have a commit somewhere that adds libmultiprocess to the guix build?

[Sjors]

Getting a cryptic error: '__NVCC___WORKAROUND_GUARD' is not defined error for the multiprocess job: https://cirrus-ci.com/task/4940413969104896?logs=ci#L2225

Somehow related to the Template Provider depending on boost/multi_index via its use of the mempool.

I can't make sense of the personality(old_personality | ADDR_NO_RANDOMIZE) error thrown by MSAN and TSAN, but it's possible that's related to my self-hosted CI setup and/or the modification in 3319668.

[ryanofsky]

re: #48 (comment)

@ryanofsky do you have a commit somewhere that adds libmultiprocess to the guix build?

I don't, since I haven't really experimented with the guix build. Maybe simplest approach to start with would enable the multiprocess build option and build and package all the multiprocess executables including bitcoin-tp and bitcoin-node. I know in the longer run you want to have separate releases as described in bitcoin#30437 (comment).

re: #48 (comment)

Getting a cryptic error: '__NVCC___WORKAROUND_GUARD' is not defined error for the multiprocess job: https://cirrus-ci.com/task/4940413969104896?logs=ci#L2225

Somehow related to the Template Provider depending on boost/multi_index via its use of the mempool.

This is a -Wundef error caused by bitcoin#29876 that happens when a boost header is included from a source file that is compiled with -I /path/to/boost/headers rather than -isystem /path/to/boost/headers. The fix for this is probably:

-bitcoin_mine_CPPFLAGS = $(bitcoin_bin_cppflags)
+bitcoin_mine_CPPFLAGS = $(bitcoin_bin_cppflags) $(BOOST_CPPFLAGS)

to add the right -isystem option.

I can't make sense of the personality(old_personality | ADDR_NO_RANDOMIZE) error thrown by MSAN and TSAN, but it's possible that's related to my self-hosted CI setup and/or the modification in 3319668.

Not sure about this error. Link seems to be https://cirrus-ci.com/task/4759864851824640?logs=ci#L2486

[Sjors]

I'm getting the MSAN / TSAN error inconsistently on my self-hosted CI. Does not seem related to this PR. Trying to reproduce in isolation in #51.

[Sjors]

New IPC release: https://github.com/Sjors/bitcoin/releases/tag/sv2-tp-0.1.14-ipc

[Sjors]

Rebased. Now that bitcoin#31741 and some other things landed, this PR is a lot simpler.

New release: https://github.com/Sjors/bitcoin/releases/tag/sv2-tp-0.1.17-ipc