accountSubscribe doesn't stream data over WS in some instances

[k-mitevski]

Problem

Hello, we are trying to figure out an issue with (so far) the methods accountSubscribe and blockSubscribe that on our nodes the notifications intermittently work.
Nodes configured exactly the same and with the same spec give different results. On one node, the notifications are streamed, while on the other(s), they are not. On some occasions, the 'bad' node will eventually start streaming, and vice versa.

We were testing out and looking into the following flags but even after modifying the values the results are still inconsistent:
rpc-pubsub-worker-threads, rpc-pubsub-notification-threads, rpc-pubsub-queue-capacity-bytes, accounts-db-cache-limit-mb.

Nodes are otherwise behaving correctly, and resource-wise, there is plenty of headroom and no resource limit hits.
Is there a limit on the parallel websocket connections that can be opened?
Or something related to the methods themselves e.g. 50 accountSubscribe channels open in parallel or similar?

To reproduce we simply open a WS connection through wscat and send one of these payloads:

{"jsonrpc":"2.0","id":1,"method":"accountSubscribe","params":["DfXygSm4jCyNCybVYYK6DwvWqjKee8pbDmJGcLWNDXjh",{"commitment":"finalized"}]}

{"jsonrpc": "2.0", "id": "1", "method": "blockSubscribe", "params": ["all", {"commitment": "finalized", "maxSupportedTransactionVersion": 0, "transactionDetails": "full", "rewards": "false"}]}

The result will get stuck and no messages will be streamed:

wscat -c ws://localhost:8800
Connected (press CTRL+C to quit)
> {"jsonrpc": "2.0", "id": "1", "method": "blockSubscribe", "params": ["all", {"commitment": "finalized", "maxSupportedTransactionVersion": 0, "transactionDetails": "full", "rewards": "false"}]}
< {"jsonrpc":"2.0","result":111870,"id":"1"}