Best practices for multiple LND instances against one bitcoind back end?

[MegalithicBTC]

A use-case for LND, or Lightning Terminal, is a multi-user setup where each user controls a separate LND or Lightning Terminal instance.

To implement this, a natural solution might be to have multiple instances of these applications (maybe running in Docker), which share a bitcoind back end.

As LND is both CPU and RAM-efficient (in my experience), it seems reasonable to think that many instances of LND could be run on one multi-core machine.

I've been investigating this issue recently and, surprisingly, I haven't yet found an established best-practice.

My first stop was LND's Install.md.

On this line

lnd/docs/INSTALL.md

Line 515 in e9fbbe2

- Don't connect more than two or three instances of `lnd` to `bitcoind`. With

... we see:

- Don't connect more than two or three instances of `lnd` to `bitcoind`. With
  the default `bitcoind` settings, having more than one instance of `lnd`, or
  `lnd` plus any application that consumes the RPC could cause `lnd` to miss
  crucial updates from the backend.

This is a direct warning against doing the thing I am proposing to do, and not only that, but "could cause LND to miss crucial updates" sounds like it might be hard to test.

So in mid July 2024 I inquired on Lightning Labs Slack about this question.

@ellemouton wrote back, with this interesting comment:

(Sorry for the screenshot. The original discussion is no longer accessible on Slack.)

So @ellemouton suggests here that, actually, the Install.md does not tell the full story... that in fact, if I reverted to using RPC polling instead, that I could run a lot of LND instances against the same bitcoind back end.

@ellemouton went on to say, in the same thread (paraphrasing here as I can no longer access the original thread), that modifying these two configuration variables...

bitcoind.blockpollinginterval
bitcoind.txpollinginterval

lnd/sample-lnd.conf

Line 782 in f0b1015

; bitcoind.blockpollinginterval=0s

... Might ALSO be a good idea. I don't think @ellemouton had a specific recommendation, but the general idea was that HIGHER values for these would REDUCE load on the bitcoind instance, and therefore be able to support more LND instances on the same machine.

I also referenced this issue: #8356
... which seems to be related to multiple LND instances against one bitcoind back end. The implications of this discussion for my query are not 100% clear to me , but I do see @Roasbeef stating, regarding txpollinterval:

I think you should be ok setting it to something on the order of 10s of minutes.

So, from this research, I decided that I should use rpc polling, and set blockpollinginterval and txpollinginterval to "some value" that might allow for a lot of LND instances.

I proceeded like this for a few weeks, but then found an issue.

When running on Testnet3 (Lightning Terminal does not support signet, so the only option is to suffer with Testnet3 currently if you want to develop with Terminal...), I found that, during "block storms" -- when dozens or hundreds of Testnet3 blocks are produced every minute, my LND nodes would regularly lose synchronization with the blockchain and become unusable.

I became suspicious that this might be happening because I had set:

bitcoind.blockpollinginterval=5s
bitcoind.txpollinginterval=5s

So I inquired about this on @alexbosworth Telegram group.

I got a very interesting response from @djkazic . Regarding the warnings in INSTALL.md, Kevin wrote:

I think this is outdated info

and...

We don't really use rpcpolling for blixt anymore due to poor performance

and..

Zmq should be fine for that with proper tuning

Now -- given the option, I'd much rather do this with ZMQ, as it seems like the more modern approach, but, on the other hand @ellemouton specifically cautioned against this exact approach.

In my mind this should be tested, and I'm looking for input on how I could test this.

If I spin up 50 LND nodes, and run them against the same bitcoind back end, how could I measure these "miss crucial updates from the backend"?

And - are there some secret ZMQ settings that I could use, which HAVE been already shown to allow a lot of concurrency with LND? If that is the route I should take?

Thanks!

[djkazic]

I would try playing with these settings:

-zmqpubhashtxhwm=n
-zmqpubhashblockhwm=n
-zmqpubrawblockhwm=n
-zmqpubrawtxhwm=n
-zmqpubsequencehwm=n

[MegalithicBTC]

@djkazic Thanks! The relevant docs for bitcoind are here: https://github.com/bitcoin/bitcoin/blob/master/doc/zmq.md

Do you have suggestions for baseline values for these that would play nice with concurrency? It just seems like this will be pretty hard to test, as I'm not even really sure how I could positively identify when the concurrency gets to be "too much" for bitcoind.

[djkazic]

The default is 1000 I think, I've set it to 10k before (and more) without harm to bitcoin core.