Increase the rebroadcast queue size

[bartolkaruza]

Problem

agave/send-transaction-service/src/send_transaction_service.rs

Line 31 in 794cb2f

const MAX_TRANSACTION_RETRY_POOL_SIZE: usize = 10_000; // This seems like a lot but maybe it needs to be bigger one day

We're seeing a lot of dropped transactions on RPC nodes, with the documented suggested approach to solving this here working as a work around, but it only amplifies the problem;

while (blockheight < lastValidBlockHeight) {
  connection.sendRawTransaction(rawTransaction, {
    skipPreflight: true,
  });
  await sleep(500);
  blockheight = await connection.getBlockHeight();
}

So if my reasoning is correct, we get dropped transactions unless we keep repeatedly spamming the rpc nodes with transactions, so that we can make it within the 10k re-broadcast queue limit at some point. This approach is a likely cause of that limit not being sufficient.

Proposed Solution

Something that might alleviate this pressure is checking if these transactions are identical and only counting unique transactions for this queue limit?

agave/send-transaction-service/src/send_transaction_service.rs

Line 31 in 794cb2f

const MAX_TRANSACTION_RETRY_POOL_SIZE: usize = 10_000; // This seems like a lot but maybe it needs to be bigger one day

[t-nelson]

it would probably be more appropriate to make this configurable from the command line than change the default behavior

[bartolkaruza]

I agree but it seems all the RPC node providers we tried seem to be running into frequent dropped transaction issues for the last months, so I imagine they are all running this same default? The default may be causing a lot of hard to diagnose issues for their users. If sufficient people try the workaround, of spamming send transactions until it gets accepted, this low default might be making the problem bigger.

We are not running an RPC node, we are running a startup that uses RPC nodes from well known providers on paid plans. So we are experiencing this issue down stream and I imagine many protocols and startups like us are experiencing the same, with the knee jerk reaction being to blame it on Solana outages/unreliability.