fix(debug-trace-server): run on jemalloc so cache churn stops inflating RSS

AGENTS.md

@@ -140,6 +140,12 @@ The memo's depth gate learns the tip from the window and `latest`-tag lookups, f
 Tag requests (`latest`/`finalized`/`safe`) bind number → hash in their single upstream header fetch instead of resolving the two separately, and every resolved (number, hash) pair — from a tag, a canonical-hash resolution, or a transaction lookup — is handed to the fetch pipeline, which then skips its own number-discovery header round trip (paid only by raw by-hash requests) and cross-checks the number against the served block's header on every tier, failing typed on a drifted pair (a cold fetch also cancels the doomed witness fetch).
 Below the response cache, a bounded in-memory `BlockData` cache keyed by block hash (`--block-data-cache-max-size`, default 1GB, 0 disables) fronts the DB and RPC tiers; block-number lookups resolve number → hash before touching it, so canonicality is never cached and it needs no reorg invalidation.
 The cache pins 4 shards (largest cacheable entry = `max_bytes / 4` on any host), counts non-retained inserts, and drops an entry when a trace fails for a data-attributable reason (`TraceError::Data` — bad witness) while request-attributable failures (invalid tracer configs) never evict.
+The binary runs on jemalloc (`tikv-jemallocator` as its `#[global_allocator]`, with `override_allocator_on_supported_platforms` so the C dependencies — zstd, aws-lc — share it) rather than the system allocator.
+On Linux, jemalloc's background purge threads are on by default (`background_threads`, enabled in a Linux-only target table because macOS jemalloc has none and only warns): without them an arena returns freed pages only while it keeps allocating, and after a load burst on a 32-core host RSS stayed ~1.1 GiB above the live heap instead of falling back to it; `MALLOC_CONF=background_thread:false` turns them off at deploy time.
+Under response- and block-data-cache churn, glibc malloc keeps freed memory in its per-thread arenas: production RSS ran at 2.5–3× the caches' accounted bytes and kept rising while the live heap stayed flat, where jemalloc under the same load tracks the accounted bytes.
+jemalloc's own heap accounting is published as `debug_trace_jemalloc_{allocated,active,metadata,resident,mapped,retained}_bytes`, sampled every 10s on a dedicated thread (`metrics::spawn_allocator_stats`) so jemalloc's stats refresh stays off the async workers; a jemalloc that cannot report statistics fails the first sample, which disables sampling with one warning.
+The effective allocator config is logged once at startup (`log_allocator_config` in `main.rs`) regardless of `--metrics-enabled`, since `MALLOC_CONF` can override the compiled-in defaults.
+RSS growth with only chain sync running is redb's page cache (`Database::create` defaults to a 1 GiB cache), bounded rather than a leak.
 In local cache mode with a `--witness-generator-endpoint` plus at least one fallback `--witness-endpoint`, request-serving witness fetches route by block age: blocks at least `--witness-local-window` blocks below the local tip skip the generator (which prunes beyond its `BACKUP` window) and fetch from the fallbacks; without the generator flag, witness endpoints are plain failover and routing is disabled.
 With the `--r2-*` flag group (endpoint, bucket, access key id, secret), every request-serving witness fetch tries a direct SigV4-signed R2 GET (light decode, capped at half the remaining witness budget) before the RPC chain, falling back on any failure; `--r2-max-concurrent-requests` caps R2 GETs separately from the RPC witness semaphore.
 `--r2-custom-domain` is the alternative R2 target (mutually exclusive with `--r2-endpoint`, rejected at startup by name): unsigned GETs of `/{key}` through a Cloudflare custom domain fronting the bucket, which negotiates HTTP/2 (many in-flight GETs multiplex over a few connections instead of holding one each against the h1.1-only S3 endpoint) and can serve the immutable witness objects from edge cache; optional `--r2-access-client-id`/`--r2-access-client-secret` attach Cloudflare Access service-token headers (rejected at startup on a non-loopback `http://` domain, and validated as header values there so a stray newline fails by name instead of becoming a per-GET retryable transport error).

Cargo.toml

@@ -107,6 +107,8 @@ sha2 = { version = "0.10", default-features = false }
 tar = { version = "0.4", default-features = false }
 tempfile = { version = "3.27", default-features = false }
 thiserror = { version = "2.0", default-features = false }
+tikv-jemalloc-ctl = { version = "0.7", features = ["stats"], default-features = false }
+tikv-jemallocator = { version = "0.7", features = ["override_allocator_on_supported_platforms"], default-features = false }
 tokio = { version = "1.51", features = ["rt-multi-thread", "signal"], default-features = false }
 tokio-util = { version = "0.7", default-features = false }
 tower = "0.5"

README.md

@@ -193,7 +193,11 @@ Size it with `--block-data-cache-max-size` (default `1GB`; `0` disables); the ca
 The byte budget is an estimate of retained payload, not exact RSS: map/allocator overhead is under-counted, and contract bytecode is charged here *and* in the contract cache (the same refcounted allocations), so evicting an entry can free less than its accounted weight.
 A cached witness is charged as its KV/level payload plus the direct-lookup table built beside it once per block and shared by every trace request.
 The table stays resident as long as the entry does, so the same budget holds fewer blocks than the witness maps alone would suggest.
-Combined default memory across the four caches is roughly 3.1 GB — response cache 1 GB + block-data cache 1 GB + contract cache 512 MB + canonical-hash memo ~0.6 GB at its 8M-entry cap (it fills lazily; the measured working set is 50–130k heights, ~10 MB) — on top of ordinary heap; size hosts (or lower the knobs) accordingly.
+The server runs on jemalloc rather than the system allocator: under glibc malloc this cache churn left RSS at 2.5–3× the caches' accounted bytes and still rising, because freed memory stayed in malloc's per-thread arenas instead of returning to the OS.
+On Linux, jemalloc runs with background purge threads, so memory freed by a load burst returns to the OS within seconds even from arenas that then go quiet; set `MALLOC_CONF=background_thread:false` to disable them.
+The `Allocator: jemalloc` line logged at startup, with or without metrics, reports the settings that took effect (`background_thread`, `narenas`, and the compiled-in `malloc_conf` that `MALLOC_CONF` overrides).
+With `--metrics-enabled`, jemalloc's own accounting is exported every 10s as `debug_trace_jemalloc_{allocated,active,metadata,resident,mapped,retained}_bytes`: `allocated` is the live heap and `resident` the physical memory the allocator holds, so a gap that widens while `allocated` stays flat is allocator retention rather than a leak.
+Combined default memory across the four caches is roughly 3.1 GB — response cache 1 GB + block-data cache 1 GB + contract cache 512 MB + canonical-hash memo ~0.6 GB at its 8M-entry cap (it fills lazily; the measured working set is 50–130k heights, ~10 MB) — on top of ordinary heap and, in local cache mode, redb's page cache (up to 1 GiB); size hosts (or lower the knobs) accordingly.
 
 **Witness endpoints:**
 Declare the internal witness generator via `--witness-generator-endpoint`; `--witness-endpoint` lists the durable fallbacks (e.g. an R2-backed witness service), tried in order.

bin/debug-trace-server/Cargo.toml

@@ -53,12 +53,19 @@ redb.workspace = true
 serde.workspace = true
 serde_json = { workspace = true, features = ["raw_value"] }
 thiserror.workspace = true
+tikv-jemalloc-ctl.workspace = true
+tikv-jemallocator.workspace = true
 tokio = { workspace = true, features = ["macros", "net", "rt", "sync", "time"] }
 tokio-util.workspace = true
 tower.workspace = true
 tower-http.workspace = true
 tracing.workspace = true
 
+# jemalloc's background purge threads exist only on Linux; elsewhere enabling them just prints a
+# warning at every startup.
+[target.'cfg(target_os = "linux")'.dependencies]
+tikv-jemallocator = { workspace = true, features = ["background_threads"] }
+
 [dev-dependencies]
 alloy-network.workspace = true
 alloy-primitives.workspace = true

bin/debug-trace-server/src/main.rs

@@ -100,6 +100,35 @@ use server_db::{BlockStore, ServerDB};
 
 use crate::chain_sync::{TraceFetcher, TraceHooks, TraceProcessor};
 
+/// jemalloc rather than the system allocator: under response- and block-data-cache churn, glibc
+/// malloc keeps freed memory in its per-thread arenas, so RSS climbs to several times the caches'
+/// accounted bytes while the live heap stays flat. `override_allocator_on_supported_platforms`
+/// routes the C dependencies (zstd, aws-lc) through it too. On Linux, `background_threads` purges
+/// freed pages off the request path; without it an arena only returns them while it keeps
+/// allocating, so memory freed by a burst stays resident in every arena that then goes quiet.
+#[global_allocator]
+static ALLOC: tikv_jemallocator::Jemalloc = tikv_jemallocator::Jemalloc;
+
+/// Logs the allocator configuration in force, whether or not metrics are enabled: `MALLOC_CONF`
+/// can override the compiled-in defaults at deploy time, and nothing else shows what took
+/// effect. A field jemalloc cannot report (background threads on macOS) is omitted.
+fn log_allocator_config() {
+    use tikv_jemalloc_ctl::{background_thread, config, opt, version};
+
+    info!(
+        version = version::read().ok().map(jemalloc_str),
+        compiled_conf = config::malloc_conf::read().ok().map(jemalloc_str),
+        background_thread = background_thread::read().ok(),
+        narenas = opt::narenas::read().ok(),
+        "Allocator: jemalloc"
+    );
+}
+
+/// Strips the NUL terminator `tikv-jemalloc-ctl`'s string reads keep from the C string.
+fn jemalloc_str(s: &'static str) -> &'static str {
+    s.trim_end_matches('\0')
+}
+
 /// Command line arguments for the debug-trace-server.
 #[derive(Parser, Debug)]
 #[clap(name = "debug-trace-server", version, about = "Debug/Trace RPC Server")]
@@ -790,6 +819,7 @@ async fn main() -> Result<()> {
         listen_addr = %args.addr,
         "Debug-trace-server starting"
     );
+    log_allocator_config();
     // The combined witness chain (generator first when configured) — the list actually fed to
     // the RPC client, not just the fallback flag values.
     let witness_apis = witness_endpoint_chain(&args);
@@ -1381,6 +1411,19 @@ fn run_prune_cycle(
 mod tests {
     use super::*;
 
+    /// Background purge threads are what return a burst's freed pages from arenas that then go
+    /// quiet, and nothing else would notice the `background_threads` feature going missing.
+    /// Linux only, like the feature. Reads the compiled-in default the feature sets rather than
+    /// the live setting, which the documented `MALLOC_CONF=background_thread:false` overrides.
+    #[cfg(target_os = "linux")]
+    #[test]
+    fn jemalloc_purges_in_the_background() {
+        let conf = jemalloc_str(tikv_jemalloc_ctl::config::malloc_conf::read().unwrap());
+        // jemalloc applies options left to right, so the last setting is the one in force.
+        let mut default = conf.split(',').filter_map(|opt| opt.strip_prefix("background_thread:"));
+        assert_eq!(default.next_back(), Some("true"), "compiled-in malloc_conf: {conf:?}");
+    }
+
     /// Verifies that an endpoint flag accepts repeated flags, CSV values, and env var —
     /// ensuring container deployments configured purely via env are not silently limited
     /// to one endpoint (clap's `value_delimiter` applies to env-var values too).

bin/debug-trace-server/src/metrics.rs

@@ -5,15 +5,17 @@
 //!
 //! Uses `metrics-derive` for declarative metric definitions following mega-reth patterns.
 
-use std::net::SocketAddr;
+use std::{net::SocketAddr, thread, time::Duration};
 
-use eyre::Result;
+use eyre::{Result, eyre};
 use metrics::{Counter, Gauge, Histogram, counter, gauge, histogram};
 use metrics_derive::Metrics;
 pub use stateless_common::{
     DEFAULT_METRICS_PORT,
     metrics::{BYTE_BUCKETS, REORG_DEPTH_BUCKETS, install_prometheus_exporter},
 };
+use tikv_jemalloc_ctl::{Access, Mib, epoch, epoch_mib, stats};
+use tracing::{debug, warn};
 
 /// Prefix for timed RPC method aliases.
 pub const TIMED_PREFIX: &str = "timed_";
@@ -746,6 +748,109 @@ impl ChainSyncMetrics {
     }
 }
 
+/// jemalloc's own accounting of the process heap (singleton), sampled by
+/// [`spawn_allocator_stats`].
+#[derive(Clone, Metrics)]
+#[metrics(scope = "debug_trace")]
+pub struct AllocatorMetrics {
+    /// Bytes allocated by the application: the live heap
+    jemalloc_allocated_bytes: Gauge,
+    /// Bytes in active pages, at least `allocated`; the excess is page-level fragmentation
+    jemalloc_active_bytes: Gauge,
+    /// Bytes dedicated to jemalloc's own metadata
+    jemalloc_metadata_bytes: Gauge,
+    /// Upper bound on bytes in physically resident pages mapped by jemalloc: active pages,
+    /// metadata, and dirty pages not yet returned to the OS
+    jemalloc_resident_bytes: Gauge,
+    /// Bytes in active extents mapped by jemalloc
+    jemalloc_mapped_bytes: Gauge,
+    /// Bytes of virtual mappings retained for reuse instead of unmapped; address space, mostly
+    /// not resident
+    jemalloc_retained_bytes: Gauge,
+}
+
+/// Publishes [`AllocatorMetrics`] from jemalloc.
+///
+/// The mallctl names are resolved to MIBs once, so a sample is seven direct lookups rather than
+/// seven walks of jemalloc's ctl tree on a loop that runs for the process's lifetime.
+struct AllocatorSampler {
+    epoch: epoch_mib,
+    stats: [(Gauge, Mib<[usize; 2]>); 6],
+}
+
+impl AllocatorSampler {
+    /// Resolves every statistic's MIB.
+    fn new() -> tikv_jemalloc_ctl::Result<Self> {
+        let AllocatorMetrics {
+            jemalloc_allocated_bytes,
+            jemalloc_active_bytes,
+            jemalloc_metadata_bytes,
+            jemalloc_resident_bytes,
+            jemalloc_mapped_bytes,
+            jemalloc_retained_bytes,
+        } = AllocatorMetrics::default();
+        Ok(Self {
+            epoch: epoch::mib()?,
+            stats: [
+                (jemalloc_allocated_bytes, stats::allocated::mib()?.0),
+                (jemalloc_active_bytes, stats::active::mib()?.0),
+                (jemalloc_metadata_bytes, stats::metadata::mib()?.0),
+                (jemalloc_resident_bytes, stats::resident::mib()?.0),
+                (jemalloc_mapped_bytes, stats::mapped::mib()?.0),
+                (jemalloc_retained_bytes, stats::retained::mib()?.0),
+            ],
+        })
+    }
+
+    /// Advances jemalloc's stats epoch and publishes one sample.
+    ///
+    /// Once the MIBs resolve, a read can only fail for a reason fixed for the process's lifetime
+    /// — chiefly a jemalloc that cannot report statistics (one linked through
+    /// `JEMALLOC_OVERRIDE`), whose stat names still resolve — so the first sample's outcome is
+    /// every sample's.
+    fn sample(&self) -> tikv_jemalloc_ctl::Result<()> {
+        // jemalloc caches its statistics; they only refresh when the epoch advances.
+        self.epoch.advance()?;
+        for (gauge, mib) in &self.stats {
+            let bytes: usize = mib.read()?;
+            gauge.set(bytes as f64);
+        }
+        Ok(())
+    }
+}
+
+/// How often [`spawn_allocator_stats`] samples jemalloc.
+const ALLOCATOR_STATS_INTERVAL: Duration = Duration::from_secs(10);
+
+/// Samples jemalloc's heap statistics into [`AllocatorMetrics`] every
+/// [`ALLOCATOR_STATS_INTERVAL`] for the process's lifetime.
+///
+/// The first sample is taken here, and a failure disables sampling with one warning rather than
+/// failing every [`ALLOCATOR_STATS_INTERVAL`]. On its own thread so the epoch refresh, which
+/// merges every arena's statistics under the arenas' locks, never runs on an async worker.
+fn spawn_allocator_stats() -> Result<()> {
+    let sampler = AllocatorSampler::new().and_then(|sampler| sampler.sample().map(|()| sampler));
+    let sampler = match sampler {
+        Ok(sampler) => sampler,
+        Err(error) => {
+            warn!(%error, "jemalloc statistics are unavailable; allocator metrics disabled");
+            return Ok(());
+        }
+    };
+    thread::Builder::new()
+        .name("jemalloc-stats".to_owned())
+        .spawn(move || {
+            loop {
+                thread::sleep(ALLOCATOR_STATS_INTERVAL);
+                if let Err(error) = sampler.sample() {
+                    debug!(%error, "Failed to sample jemalloc stats");
+                }
+            }
+        })
+        .map_err(|e| eyre!("failed to spawn the jemalloc stats thread: {e}"))?;
+    Ok(())
+}
+
 /// Inbound admission-gate occupancy, labeled `(method)`.
 ///
 /// Two independent gauges rather than a queued/executing pair, because the two phases are
@@ -1029,6 +1134,8 @@ pub fn init_metrics(addr: SocketAddr) -> Result<()> {
     // Pre-register all metrics
     pre_register_all_metrics();
 
+    spawn_allocator_stats()?;
+
     Ok(())
 }
 
@@ -1235,6 +1342,44 @@ mod tests {
         assert_eq!(resolve_method("nonexistent"), "unknown");
     }
 
+    /// The gauges only carry real numbers when jemalloc is the allocator this binary runs on
+    /// and can report statistics; otherwise sampling is disabled and the dashboard shows nothing.
+    ///
+    /// Asserts presence and non-zero values only, not orderings such as
+    /// `allocated <= active`: jemalloc merges the per-arena counters without a global lock, so
+    /// a snapshot taken while other tests allocate in parallel can briefly violate them.
+    #[test]
+    fn allocator_metrics_sample_the_live_heap() {
+        use std::collections::HashMap;
+
+        use metrics_util::debugging::{DebugValue, DebuggingRecorder};
+
+        let recorder = DebuggingRecorder::new();
+        let snapshotter = recorder.snapshotter();
+        metrics::with_local_recorder(&recorder, || {
+            AllocatorSampler::new().unwrap().sample().unwrap()
+        });
+
+        let gauges: HashMap<String, f64> = snapshotter
+            .snapshot()
+            .into_vec()
+            .into_iter()
+            .filter_map(|(key, _, _, value)| match value {
+                DebugValue::Gauge(v) => Some((key.key().name().to_owned(), v.into_inner())),
+                _ => None,
+            })
+            .collect();
+        // The recorder sees the scoped name; the Prometheus exporter renders `.` as `_`.
+        let gauge = |stat: &str| gauges[&format!("debug_trace.jemalloc_{stat}_bytes")];
+
+        // The test harness itself allocates, so every stat but `retained` (address space, which
+        // jemalloc may hold none of) is non-zero in any process.
+        for stat in ["allocated", "active", "metadata", "resident", "mapped"] {
+            assert!(gauge(stat) > 0.0, "{stat}");
+        }
+        assert!(gauges.contains_key("debug_trace.jemalloc_retained_bytes"));
+    }
+
     #[test]
     fn test_timed_aliases_consistency() {
         for &(alias, _original) in TIMED_METHOD_ALIASES {