ElectrumX server doesn't support fee estimation

[nkuba]

ElectrumX server instance we're running failed integration tests for fee estimation:

➜  keep-core git:(electrum-wss) ✗ go test -v -tags=integration ./pkg/bitcoin/electrum -run TestEstimateSatPerVByteFee_Integration
=== RUN   TestEstimateSatPerVByteFee_Integration
=== RUN   TestEstimateSatPerVByteFee_Integration/electrumx_tcp
=== RUN   TestEstimateSatPerVByteFee_Integration/electrumx_wss
    electrum_integration_test.go:439: daemon does not have enough information to make an estimate
=== RUN   TestEstimateSatPerVByteFee_Integration/fulcrum_tcp
=== RUN   TestEstimateSatPerVByteFee_Integration/electrs-esplora_tcp
=== RUN   TestEstimateSatPerVByteFee_Integration/electrs-esplora_ssl
--- FAIL: TestEstimateSatPerVByteFee_Integration (3.10s)
    --- PASS: TestEstimateSatPerVByteFee_Integration/electrumx_tcp (0.46s)
    --- FAIL: TestEstimateSatPerVByteFee_Integration/electrumx_wss (0.79s)
    --- PASS: TestEstimateSatPerVByteFee_Integration/fulcrum_tcp (0.82s)
    --- PASS: TestEstimateSatPerVByteFee_Integration/electrs-esplora_tcp (0.46s)
    --- PASS: TestEstimateSatPerVByteFee_Integration/electrs-esplora_ssl (0.56s)
FAIL
FAIL    github.com/keep-network/keep-core/pkg/bitcoin/electrum  3.377s
FAIL

We need to investigate and fix the problem.

Pull Requests

Preview Give feedback

1. Kubernetes configuration for bitcoin and electrumx #3573
   ☁️ infrastructure +1
   

[nkuba]

We are running ElectrumX server with lukechilds/electrumx Docker image which is based on spesmilo/electrumx server implementation.
The ElectrumX server is backed by a Bcoin node that we're also running.

The estimatefee method implementation in the ElectrumX server calls estimatesmartfee method on the Bitcoin RPC and extracts feerate property (https://github.com/spesmilo/electrumx/blob/32a149500ebbfb67249c00160ad3f928627c2126/electrumx/server/daemon.py#L259-L261).
This is consistent with Bitcoin Core RPC API (https://developer.bitcoin.org/reference/rpc/estimatesmartfee.html#result).

Unfortunately, Bcoin RPC API is inconsistent with the Bitcoin Core RPC API, and instead of returning the estimate under feerate property it outputs a fee property (https://bcoin.io/api-docs/#estimatesmartfee). This makes the ElectrumX server fail the value fetch.

I reported the problem in Bcoin repo (bcoin-org/bcoin#1153).

We should consider switching Bcoin to Bitcoin Core as our full node. According to Fulcrum documentation.

What is interesting Bitcoin Core is specified as the only full node that is supported (https://github.com/cculianu/Fulcrum#requirements) by Fulcrum which is an alternative Electrum server implementation we may consider running in the future.

We used to run Bitcoin Core node (bitcoind which is still running in the keep-test cluster) but we switched to Bcoin for V1's relay maintainer #1709.

Resources:

• lukechilds/electrumx: https://github.com/lukechilds/docker-electrumx
• spesmilo/electrumx: https://github.com/spesmilo/electrumx
• bcoin: https://github.com/keep-network/bcoin

[nkuba]

After switching ElectrumX server to use Bitcoin Core the initial problem was solved - please see passing integration tests in #3589.

We need to update the production setup to use the Bitcoin Core.

[nkuba]

I updated bitcoin and electrum servers configurations to use Bitcoin Core. #3573 contains changes.

The #3599 PR adds integration tests that will help monitor the health of the servers.