Skip to content

perf: concurrent txpool fetcher with separate IPC connection#906

Open
vbuilder69420 wants to merge 2 commits intoflashbots:developfrom
vbuilder69420:perf/concurrent-txpool-fetcher
Open

perf: concurrent txpool fetcher with separate IPC connection#906
vbuilder69420 wants to merge 2 commits intoflashbots:developfrom
vbuilder69420:perf/concurrent-txpool-fetcher

Conversation

@vbuilder69420
Copy link
Copy Markdown

Summary

Fix the txpool subscription dying under high transaction load by:

  1. Using a separate IPC connection for get_raw_transaction_by_hash RPC calls
  2. Fetching transactions concurrently (64 parallel) instead of sequentially
  3. Increasing the subscription broadcast channel from 16 to 16,384 items

Root Cause

The previous implementation had three cascading bottlenecks:

Sequential per-tx RPC calls block the subscription consumer — For each tx hash notification, rbuilder made a blocking get_raw_transaction_by_hash call over the same IPC connection before reading the next notification. At 500+ TPS, the subscription produced hashes faster than rbuilder could fetch full txs one-by-one.

Broadcast channel of 16 items — The alloy PubSubFrontend defaults the broadcast channel to 16 items. At 500+ TPS, this overflows instantly and notifications are silently dropped (SubscriptionStream ignores Lagged errors).

No backpressure — The unbounded channels between the IPC backend and PubSub service meant the kernel socket buffer could fill up, causing reth to reset the connection.

Before / After

Metric Before After
IPC survival at 500 TPS ~5 blocks (~30s) Full run (46+ blocks)
IPC survival at 2000 TPS ~2 blocks (~10s) 5+ minutes (52 blocks)
Max orders in one block ~100 616

Design

Before:
  subscription stream → [hash] → get_raw_tx (BLOCKS) → [hash] → get_raw_tx (BLOCKS) → ...
                         ↑ same IPC connection for both ↑

After:
  IPC conn 1: subscription stream → [hash] → spawn task → [hash] → spawn task → ...
  IPC conn 2:                        ↓                      ↓
                                   get_raw_tx            get_raw_tx    (up to 64 concurrent)
                                     ↓                      ↓
                                   → orderpool           → orderpool

Test plan

  • Builds cleanly
  • IPC survives full run at 500 TPS (was dying after ~5 blocks)
  • IPC survives 5+ minutes at 2000 TPS (was dying after ~2 blocks)
  • Existing test test_fetcher_retrieves_transactions preserved
  • Integration tests pass

🤖 Generated with Claude Code

vbuilder69420 and others added 2 commits March 21, 2026 22:55
…klist check

Remove the AccessListInspector entirely from RBuilderEVMInspector.
Replace the per-opcode blocklist tracking with a post-execution check
against ResultAndState.state (EvmState = HashMap<Address, Account>),
which already contains every address touched during EVM execution.

The AccessListInspector called step() on every EVM opcode to build an
access list, solely used to check addresses against the blocklist.
Profiling showed this inspector overhead consumed ~52% of CPU time.
The EVM execution result already contains the same information in its
state diff, making the inspector entirely redundant.

Changes:
- order_commit.rs: Use create_evm() (NoOpInspector) when no
  used_state_tracer is needed. Check blocklist via res.state.keys()
  after execution instead of via access list.
- evm_inspector.rs: Remove AccessListInspector from
  RBuilderEVMInspector. The inspector now only wraps the optional
  UsedStateEVMInspector (used by parallel builder / EVM caching).

This optimization works regardless of whether a blocklist is configured.

Benchmark (builder-lab, 100 TPS, seed=42, 60s profiling window):

| Metric              | Before   | After    | Change |
|---------------------|----------|----------|--------|
| Block fill p50      | 96.8ms   | 58.9ms   | -39%   |
| Block fill p95      | 129.2ms  | 87.1ms   | -33%   |
| E2E latency p50     | 98ms     | 61ms     | -38%   |
| E2E latency p95     | 134ms    | 92ms     | -31%   |
| Blocks submitted    | 255      | 342      | +34%   |
| Txs included        | 17,882   | 23,449   | +31%   |

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The txpool fetcher previously processed tx hash notifications sequentially:
for each hash, it made a blocking get_raw_transaction_by_hash RPC call over
the same IPC connection before consuming the next notification. At high TPS,
this caused the subscription to back up and the IPC connection to die.

Three changes:
1. Use a SEPARATE IPC connection for RPC calls, so fetches don't block
   the subscription stream consumer
2. Fetch transactions CONCURRENTLY using tokio::spawn with a semaphore
   (64 concurrent fetches) instead of sequential per-hash blocking
3. Increase the subscription broadcast channel from 16 to 16384 items
   to prevent silent notification drops under load

Before: IPC subscription dies after ~5 blocks at 500 TPS
After: IPC subscription survives the full run (46+ blocks at 500 TPS,
       5+ minutes at 2000 TPS)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant