[faresobeid]

Stacked on #3902 (which sits on #3900).

What

Turns on passing router-replay ids and sampling masks by handle on multi-node runs, and owns the payload directory's lifecycle. Inline stays everywhere else.

• Switch (one config field): orchestrator.payload_root. rl sets it to <run dir>/payloads (absolute) when router replay or sampling-mask capture is on, the deployment is multi_node, and inference is not disaggregated. Inference needs no config: the directory travels with each request. Setting it by hand with disaggregated P/D is rejected, because the PD router does not stitch by-handle routing segments.

• Request: the dispatcher gives each live-sourced train rollout payload_dir = <payload_root>/v<k>, where k is the rollout's own dispatch policy version. That is never older than its group's start, which the staleness sweep keys on, so the cleanup bound below holds. Env._sampling puts it in extra_body.extra_args; verifiers flattens extra_body into the generate sampling_params and renderers forwards them, and vLLM's GenerateRequest parses it into sampling_params.extra_args["payload_dir"].

• Writer (serving_tokens.py, on top of perf(inference): pack /generate per-token payloads and tune API-server GC #3900's _PackedOutputs): for a request with payload_dir, routed_experts and the packed sampling_mask are taken off the engine output (upstream encodes nothing inline), written into one file <payload_dir>/<uuid>.bin in a thread, closed before responding (NFS close-to-open), and returned as payload segments on the choice. completion_logprobs stays packed inline.

  • Routing rows are uint8/uint16 (compact_routed_experts, split out of serialize_routed_experts) and anchored at the end: the segment covers [prompt_len + completion_len - 1 - rows, prompt_len + completion_len - 1), i.e. it ends at the last forwarded token. On a colocated server that equals routed_experts_prompt_start. Under 0.31 P/D (feat(inference): router replay on the V2 runner everywhere #3830) a decode instance emits rows only from its first forward (prompt_len - 1), so end-anchored positions stay correct there, and router#60 can clip decode rows by position.
  • Mask row i is completion token i at prompt_len + i: int32 [tokens, widest row] padded with -1, built from perf(inference): pack /generate per-token payloads and tune API-server GC #3900's CSR PackedSamplingMask.
  • Requests without payload_dir (eval, frozen-source rollouts, standalone clients) keep perf(inference): pack /generate per-token payloads and tune API-server GC #3900's inline path.
  • Shared-FS read-after-write: the writer fsyncs the file (and the v<k> directory when it creates it) before responding; the trainer's reader reopens the file and retries a short read 4 times with backoff (0.1 s → 0.8 s), then raises with the file, offset and byte counts.

• Orchestrator (trajectories.py): trace_to_samples takes branch.payload (verifiers#2765), clips it to the sample and adds a 1-row segment for the final token that points at the previous row's bytes (what the inline path's placeholder copies). A branch with any other routing gap drops its routing with a warning, like Branch.routed_experts returning None.

• Retention: delete on consume. A file belongs to one trace (verifiers attributes a request's segments to that trace only), and a trace's samples ship together in one batch, so no per-file refcount is needed.

  • TrainSink.process_batch puts the files of every selected trace (shipped or pruned at batch time) on TrainBatch.payload_files; finalize_train_batch records them under the batch's step after sender.send (an empty or over-max_steps batch deletes them at once).
  • Orchestrator.on_policy_update(step), after inference applies v{step}, deletes the files of every recorded batch ≤ step: the trainer reads batch s in full (DataLoader.get_batch) before it publishes v{s}.
  • So the directory holds only rollouts in flight, waiting for their group, queued, or shipped and not yet read, whatever max_off_policy_steps is. Before, every trained file stayed until its v<k> aged out, i.e. max_off_policy_steps + 2 steps of payloads (~100 GB at GLM-4.5-Air SWE batch 512, max_off_policy_steps = 32; ~5 TB for GLM-5-class at batch 4096 × 30k tokens, max_off_policy_steps = 64).
  • Backstop for files no batch claims (episodes cancelled or failed in flight, files from before a resume): v<k> is still removed for k < min_fresh_version(step + 1, max_off_policy_steps); batch b ≥ step + 1 ships only rollouts whose group started at min_fresh_version(b) or later, and a rollout's k is at least its group's start.
  • Deletion runs on the default executor and is not awaited: on_policy_update runs under the weight watcher's update_lock, and unlinking a step's files on a shared filesystem takes 10-35 s at batch 512 (below), which would otherwise delay the next weight update.
  • A fresh run wipes the root; a resumed run keeps it (rl cleans future batches on resume, so nothing there is read again; the backstop removes it).

• Discarded rollouts: TrainSink.process_group deletes the files no queued sample points into (dropped groups, untrainable branches, pruned samples); _drop_stale deletes the files of queued traces it sweeps.

• Docs: docs/inference.md § Payloads by handle.

Measured (GLM-4.5-Air SWE, batch 512 = 32 × 16, router replay on, 1 inference + 1 trainer node, CP 4, max_off_policy_steps = 32, 8 steps)

Same tree, config and nodes as the earlier run of this chain's previous head; this run adds the concurrent reads (#3902) and delete-on-consume. Payload root on the shared /home filesystem (virtiofs, the only shared filesystem on the nodes).

	previous head	this head
trainer time/load_data, steps 3-8 (s)	6.4 / 20.6 / 14.3 / 43.8 / 15.0 / 14.0 (median 14.6)	2.1 / 2.6 / 2.7 / 11.2 / 4.2 / 14.4 (median 3.5)
payload directory, peak	all files written (20.1 GB after 8 steps; no v<k> reaches the 32-version bound), ~90-100 GB steady state	7.4 GB / 36k files (~2.7 batches)
deletion per step	rmtree of v<k> awaited under update_lock (from step 34 on)	5.3k-15k files, 1.0-2.8 GB, 10.7-34.2 s of unlinking, in the background

Mean staleness 0.6-0.9 (max 3), as before; no read errors, no stale drops. 84.6k files (16.0 GB) were deleted on consume during the run.

Pins

Both pins combine two open companion PRs each, on pin branches:

• deps/verifiers → 5b432ddf (pin/payload-by-handle-packed-da75527): verifiers#2765's two commits on top of verifiers#2778's head (cef65c30, on verifiers main da75527).
• deps/renderers → 0c60f6c (pin/payload-passthrough-packed-41f9290): renderers#171's commit on top of renderers#174's head (19c558f, on renderers main 41f9290).

Re-pin to the merged commits once those PRs land.

Tests

• test_payload_dir_writes_routing_and_masks_by_handle drives the real _PackedOutputs iterator: routing and mask go into one file with end-anchored routing positions, CSR masks are padded with -1, and the inline fields are cleared.
• test_read_file_retries_short_reads: a short read is retried after the file is reopened, then raises.
• pytest tests/unit/orchestrator tests/unit/test_configs.py tests/unit/inference tests/unit/train/rl -m "not gpu" on vLLM 0.31.0 with the pinned verifiers/renderers on the path, ruff check, ruff format --check: pass.
• GPU (vLLM 0.31.0, Qwen3-30B-A3B TP1, batch invariant, 16 requests of 500-8000 prompt tokens, conc 8), run on the writer before it moved onto perf(inference): pack /generate per-token payloads and tune API-server GC #3900's format: 16/16 file rows equal inline routing and masks; response size 33.9 MB → 0.57 MB; trainer tensors from those rows 140/140 bitwise equal to inline. Not re-run on this head.
• End to end: the 8-step GLM-4.5-Air run above (2 nodes).