[flyq]

Summary

Switch debug-trace-server from glibc malloc to jemalloc so that response- and block-data-cache churn no longer inflates RSS.
On Linux, jemalloc's background purge threads are also turned on, so memory freed by a load burst goes back to the OS.
jemalloc's own heap accounting is exported as metrics, so live heap and allocator-retained memory can be told apart on dashboards.

In production on ORE, with both caches full and evicting, RSS now sits at 1.02× the caches' accounted bytes (36.75 GiB vs 35.88 GiB) and has stayed flat for about 1.5 days.
The glibc build on the same host reached 2.53× (90.9 GiB vs 36.0 GiB) and was still climbing.

Under cache churn, glibc malloc keeps freed memory in its per-thread arenas instead of returning it to the OS.
In production, RSS sat at 2.5–3× the caches' accounted bytes (ORE 90 GiB vs 36.5 GiB, FRA 85 GiB vs 28.5 GiB) and kept rising, while the live heap stayed flat.
A side-by-side lab run on v2.0.18 used the same binary and the same load, with only the allocator swapped (jemalloc 5.3.0 via LD_PRELOAD): 512 MB response cache plus 512 MB block-data cache, churned over 12k historical blocks for about 470k executions each, at a ~57% hit rate.

Allocator	RSS	Cache accounted	RSS − cache	Trend
glibc	1974 MiB	1017 MiB	+957 MiB	still rising
jemalloc	1043 MiB	1018 MiB	+25 MiB	flat from phase 3

Heap profiling ruled out a code leak.
Cache hits, transaction execution, R2 witness fetches and connections all hold steady.
The only live-object growth comes from chain sync, and it is redb's page cache, which is bounded.

Changes

Allocator

• Make tikv-jemallocator 0.7 (jemalloc 5.3.1, the same version mega-reth uses) the binary's #[global_allocator].
• Enable override_allocator_on_supported_platforms so the C dependencies (zstd, aws-lc) also allocate through jemalloc on Linux.
• No cfg(unix) gating: the binary is already unix-only (timing.rs calls clock_gettime(CLOCK_THREAD_CPUTIME_ID) ungated).
• Every start logs one Allocator: jemalloc line with the jemalloc version, the compiled-in malloc_conf, background_thread and narenas, whether or not metrics are enabled, so a MALLOC_CONF override shows up in the logs.

Background purge threads (Linux)

• Enable background_threads in a Linux-only target table.
  macOS jemalloc has no background threads and only prints a warning at every startup, so the feature is not enabled there.
• Without these threads, an arena returns freed pages only while it keeps allocating.
  So memory freed by a burst stays resident in every arena that then goes quiet; a 32-core host has 128 arenas.
• To turn them off at deploy time, set MALLOC_CONF=background_thread:false.
• A Linux-only unit test, jemalloc_purges_in_the_background, pins the feature.
  It reads the compiled-in malloc_conf rather than the live setting, so a MALLOC_CONF=background_thread:false environment does not fail it.

Metrics

• With --metrics-enabled, sample jemalloc's stats.{allocated,active,metadata,resident,mapped,retained} every 10s into debug_trace_jemalloc_*_bytes gauges, the same set mega-reth exports.
  This adds tikv-jemalloc-ctl with its stats feature, which builds jemalloc with statistics enabled.
• allocated is the live heap and resident the physical memory jemalloc holds, so a gap that widens while allocated stays flat is allocator retention rather than a leak.
• The stat lookups are resolved once, and the first sample runs at startup.
  If jemalloc cannot report statistics, sampling is disabled with a single warning instead of failing every 10s.
• Sampling runs on a dedicated thread so the epoch refresh, which merges every arena's stats under their locks, stays off async workers.
  The /metrics exporter itself still runs on the main runtime, so this does not keep the gauges readable while tracing saturates it.

Docs

• README and AGENTS.md cover the allocator choice, the background threads, the startup log line and the new gauges.
• They note that RSS growth with only chain sync running is redb's page cache (Database::create defaults to 1 GiB): bounded, not a leak.

Testing

Local

• cargo fmt --all --check, cargo sort --check … and cargo clippy --workspace --all-targets --all-features pass with no warnings.
• cargo test -p debug-trace-server: all 207 unit tests pass on macOS.
  allocator_metrics_sample_the_live_heap guards the stats feature and jemalloc being the active allocator.
• A local run shows the Allocator: jemalloc startup line and all six debug_trace_jemalloc_*_bytes gauges on /metrics.

tmp-0 (Ubuntu 24.04, release build, --locked)

• Linux allocator override is verified.
  The binary exports malloc/free/calloc/realloc, and LD_DEBUG=bindings shows glibc's own malloc/free binding to them.
  Four jemalloc_bg_thd threads start without any MALLOC_CONF.
• Correctness against a mega-reth v2.2.1 full node, using the same normalization as consistency_test.rs:
  • Scope: 1000 blocks (23,073 txs) served from the local DB, plus 100 near-tip blocks whose witnesses came from R2.
  • callTracer, trace_block, prestateTracer, by-hash, debug_traceTransaction and trace_transaction were all identical, with no dts errors.
  • Remaining differences: 4byteTracer (mega-reth answers with struct logs; the gateway rejects this tracer anyway), prestateTracer diffMode (empty {} post entries, same in v2.0.20), and the index / "refund":0 fields that chore: upgrade to reth v2.3.0 / revm 40 / alloy 2.0.5 #184 already brought to main.
• Memory under load, measured on the v2.0.20-based build that carries these same commits.
  Each run was ~500k callTracer plus 50k prestateTracer requests over 1000 cached blocks, at 25–37k req/s, with 0 errors:

	Live heap (allocated)	RSS after a 2-minute idle
No background threads	~630 MiB	~1.8 GiB
background_threads	~620 MiB	~0.7 GiB

Production (ORE)

mnet-ore-compute-0: 48 cores, 182 GiB RAM, no swap, shared with rpc-node (48.4 GiB RSS).
Once full, the response cache sits just under 32 GiB and the block-data cache just under 4 GiB.
Snapshots below were taken from /proc, /metrics and debug_getCacheStatus (times in UTC):

Run	Allocator	Snapshot	RSS	Caches accounted	RSS − caches	Cache state
until 10/16 14:28	glibc	10/16 14:28	90.9 GiB	36.0 GiB	+54.9 GiB (2.53×)	both full; RSS still rising
10/16 14:28 → 10/19 09:03 (restart only)	glibc	10/19 09:00, up 2d18h	22.6 GiB	21.9 GiB	+0.76 GiB	response cache still filling (17.9 GiB)
since 10/19 09:03 (d636173)	jemalloc + background threads	10/22 09:27, up 3d0h	36.75 GiB	35.9 GiB	+0.88 GiB (1.02×)	both full since ~10/20 17:00

• In the last two runs RSS grew linearly while the caches filled, and stayed within 1 GiB of the caches' accounted bytes the whole time.
  The growth is the caches filling, not retention.
• The glibc run between 10/16 and 10/19 shows no inflation because its response cache never filled, so it evicted nothing.
  That fits glibc's growth coming from eviction churn, which only starts once a cache is full.
• Around 10/20 16:30–18:00, a burst of cache-filling traffic raised RSS by about 16 GiB in under two hours, up to the response cache's limit.
  Since then RSS has stayed flat at about 36.7 GiB for roughly 39 hours, while both caches evict.
• jemalloc's own accounting at the latest snapshot: allocated 35.2 GiB, active 36.5 GiB, resident 36.9 GiB, metadata 0.33 GiB, mapped 37.1 GiB.
  resident is 1.67 GiB (4.7%) above allocated.
  retained (22.1 GiB) is address space kept for reuse, not resident memory.
• Caveat: the jemalloc run has seen less churn than the glibc run had by 10/16.
  It has served 1.11M requests, with 517k response-cache and 509k block-data misses, against 10.18M requests and 3.17M / 3.11M misses for the glibc run.

Rollout

• ORE runs liquan/fix/dts-jemalloc-v2.0.20 at d636173: v2.0.20 plus this PR's allocator, metrics and background-thread commits, without chore: upgrade to reth v2.3.0 / revm 40 / alloy 2.0.5 #184 and without the later sampler and startup-log cleanups.
• FRA stays on v2.0.20 (glibc) as the control.
• Both report version 2.0.20; only ORE exposes debug_trace_jemalloc_*.

🤖 Generated with Claude Code

[mega-maxwell]

Claude review status

Living comment — rewritten in place. The review workflow keeps this single comment up to date instead of posting a new one each round, so it always describes the latest reviewed commit and the earlier text is intentionally gone. No reply is needed here; reply to a finding in its own review thread, and answer an open question in a reply on this PR. The next review round reconciles your answer.

🛠️ Review did not finish

Attempted bac73c29..7937fec6 · updated 2026-10-22T09:11:36+00:00

This round did not publish: MODEL_SESSION_LIMIT: Claude session limit reached (5-hour quota).. Anything listed below is from the last round that did. Workflow run: https://github.com/megaeth-labs/stateless-validator/actions/runs/39753317204