issue with greenlight node connecting to something

[imaginator]

As mentioned in breez/breez-sdk-greenlight#1158, we see errors connecting via the breez SDK that looks like a greenlight issue.

https://gist.github.com/imaginator/bf504ca536d5b3e8f7685f94e8be3e87 is a full log where it looks like the greenlight node is having issues connecting to a plugin.

Received log [ INFO ]:  node-logs: UNUSUAL plugin-gl-plugin-internal: JSON-RPC interface not yet available. Delaying 50ms. IoError(Os { code: 2, kind: NotFound, message: \"No such file or directory\" })

Please let me know what other info I can send to help debug.

[imaginator]

Update: it appears that this JSON-RPC interface not yet available every 50ms warning has resolved and we are now able to create invoices again.

I'd still be curious to understand where this error was coming from though, especially as we get closer to rolling our service out to users.

[cdecker]

@imaginator the root cause for this issue is that the node ended up being scheduled in a host that was already under heavy load. This means that startup can take considerable time, exceeding the 10s we allow for startup. RPC calls are held for up to those 10s, but let through if the timeout expires, which then results in the JSON-RPC socket file not having been created yet due to the slow startup.

It's a pure server side issue, and we need to solve for it on the servers. I saw your report, and it caused me to add a bunch of capacity to our servers, but I must've forgotten to add a comment here. Apologies 🙏