perf(inference): pack /generate per-token payloads and tune API-server GC by faresobeid · Pull Request #3900 · PrimeIntellect-ai/prime-rl

faresobeid · 2026-10-06T13:39:48Z

Problem

Under RL load (GLM-4.5-Air, 1 trainer + 1 inference node, ~130-300 concurrent 16k-token requests), the vLLM API server spends 25% (top_p 1.0) to 42-43% (top_p 0.97 + sampling replay) of its wall time in gen-2 GC, with single pauses up to 64 s. During a pause the server answers nothing (/health, /metrics, completions, weight-update calls). With sampling replay this cost -30% effective generation throughput and +50% step time, and with default router health checks 25-83% of rollouts failed.

Cause: tens of millions of GC-tracked per-token objects alive at once in the API-server process: Logprob objects and dicts per token plus decoded strings (output processor), one pydantic object per token in _create_tokens_logprobs, one Python list per token for the sampling mask, and the JSON encoding of all of it. Every gen-2 pass walks them all.

Change

/inference/v1/generate (prime-rl's PrimeRlServingTokens) now builds no per-token objects:

Packed payloads, encoded like routed_experts (one encode_array helper for all three; routed_experts.py is removed). routed_experts adds start, anchored at the last forwarded token: len(prompt) + len(completion) - 1 - len(rows). Without P/D this equals routed_experts_prompt_start; on a P/D decode instance (feat(inference): router replay on the V2 runner everywhere #3830) the rows begin at its first forward.
- choice.completion_logprobs = {data, shape, dtype}: base64 float32, the sampled token's logprob per completion token (clamped at -9999 like upstream). choice.logprobs is no longer set, so /generate no longer returns top_logprobs (whatever sampling_params.logprobs asks for). renderers asks for logprobs = 1 and only reads the sampled logprob, so nothing in our stack uses them.
- choice.sampling_mask = {"ids": {...}, "counts": {...}}: CSR base64 int32 arrays instead of list[list[int]].
Flat logprobs, no detokenization: non-streaming /generate requests get flat_logprobs = True (vLLM's FlatLogprobs: a few lists per request instead of a dict + Logprob objects per token) unless they ask for prompt logprobs, and detokenize = False unless they set stop strings (/generate returns token ids only, so the text and the per-token decoded logprob strings were never used).
Masks stay numpy in the output processor (monkey_patch_packed_sampling_masks, non-streaming requests only): keep only the per-step id arrays (untracked by GC) instead of a tuple per token, and hand the serving layer a PackedSamplingMask(ids, counts) instead of list[list[int]].
API-server GC: after app init, gc.collect(); gc.freeze() (startup objects leave GC) and gc.set_threshold(50_000, 20, 100) (default (700, 10, 10)).
Streaming /generate and the OpenAI endpoints are unchanged.

Clients: PrimeIntellect-ai/renderers#174 reads completion_logprobs (same checks as the per-token path); PrimeIntellect-ai/verifiers#2778 decodes the packed mask. Both keep the old forms for stock vLLM.

Pins

deps/renderers 41f9290 -> ca029b5 (renderers main, the merge commit of renderers#174).
deps/verifiers 2cb0a6d -> e888a71 (verifiers main, the merge commit of verifiers#2778).

Landing order: renderers#174 -> verifiers#2778 -> this PR, with the prime-rl re-pins to renderers/verifiers main in the same commit as the server change. An old client against a new server gets no logprobs.content.

Measurement

One node, GLM-4.5-Air TP8, 130 concurrent closed-loop requests (Polaris prompts, top_p 0.97 / top_k 512, max 16k tokens, ~12.5k tokens/response), 15 min per arm, before (main) vs after (this branch):

arm	gen tok/s	done reqs	gen2 GC % wall	all GC % wall	gen2 count	max GC pause s	health p50/p99/max s	resp MB mean	client parse ms mean	tokens/resp
base-nomask	5497	512	11.8	13	164	1.87	0.003/1.34/21.7	1.71	28.5	12490
fix-nomask	5601	516	0	0	0	0	0.003/0.01/0.3	0.12	1	12619
base-mask	5453	508	23.8	25.8	111	4.27	0.003/4.37/30.1	1.83	31.4	12479
fix-mask	5446	504	0	0.9	0	0.99	0.003/0.07/1.8	0.31	1.4	12582
fix-mask-nogctune	5480	512	5.1	5.7	31	2.63	0.003/1.94/2.8	0.31	1.3	12500

The closed-loop tok/s does not move, because the engine core keeps generating during API-server pauses. The RL cost came from what the pauses block (health checks, dispatch, responses, weight updates), measured here by /health latency. Ablation: packing alone takes gen-2 GC from 23.8% to 5.1% of wall time; gc.freeze + thresholds take it to zero gen-2 passes.

Note

Medium Risk
Changes the /generate response contract and patches vLLM internals; clients must understand packed fields, though streaming and OpenAI routes are unchanged.

Overview
Under heavy RL concurrency, /inference/v1/generate no longer builds millions of per-token Python objects in the API server. Non-streaming responses now return packed, base64 numpy payloads for routed_experts, new completion_logprobs (sampled-token logprob per position), and CSR sampling_mask (ids + counts), with serialization consolidated in serving_tokens and routed_experts.py removed.

The handler enables flat logprobs and disables detokenization when safe (no prompt logprobs / no stop strings), so upstream stops materializing Logprob dicts and decoded strings. A vLLM output-processor monkey patch keeps sampling-mask chunks as numpy until finish for FINAL_ONLY requests.

After app startup, the API server runs gc.collect() / gc.freeze() and raises GC thresholds to cut gen-2 pauses. Unit tests cover packing and post_process; renderers is pinned to a release that reads the new fields (paired client PRs).

^{Reviewed by Cursor Bugbot for commit a6f0da5. Bugbot is set up for automated code reviews on this repo. Configure here.}

cursor

Stale Bugbot comment from a previous run.

cursor

Stale Bugbot comment from a previous run.

cursor

Stale Bugbot comment from a previous run.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit bb77d68. Configure here.}

…r GC Return completion logprobs and sampling masks as base64 numpy arrays (like routed_experts), keep masks as numpy in the output processor, request flat logprobs and skip detokenization on /generate, and gc.freeze() plus higher GC thresholds in the API server. Bumps renderers/verifiers to the companion branches that parse the packed fields.

Streaming (DELTA) and cumulative outputs keep vLLM's chunk list and SamplingMask form; only FINAL_ONLY requests, which PrimeRlServingTokens reads, get the packed CSR arrays. Document that each mask chunk is one position.

…orwarded token start = prompt + completion - 1 - rows. Equals routed_experts_prompt_start on a colocated instance; on a P/D decode instance the rows begin at its first forward.

…ked /generate path The start moves into _PackedOutputs (#3900), so routed_experts.py and serving_tokens.py stay as on main.

… main

…2778) `SamplingMask.from_sampling_mask` now also accepts prime-rl's packed CSR form. prime-rl's `/inference/v1/generate` server now returns sampling masks as `{"ids": {data, shape, dtype}, "counts": {data, shape, dtype}}` (base64 int32 arrays) instead of `list[list[int]]`. The one-list-per-token form was a large part of the vLLM API server's GC load and JSON size under RL traffic (see the prime-rl PR). Here it also skips building one Python list per token when the client parses the response. - dict input: decode `ids` and `counts` straight into the existing int32 arrays (`_decode_packed`, which #2756 reuses for its packed logprobs). - list input (stock vLLM): unchanged. Companions: renderers PrimeIntellect-ai/renderers#174 / prime-rl PrimeIntellect-ai/prime-rl#3900. Land renderers -> verifiers -> prime-rl; the prime-rl submodule pins must move to the merged commits. ## Validation - `tests/v1/test_trace.py tests/v1/test_graph.py`: pass (one new test: packed and list inputs give the same arrays). - `ruff check` / `ruff format` (0.16.0) on the changed files: clean.  --- > [!NOTE] > **Low Risk** > Additive parsing path with existing list behavior preserved; main caveat is packed `shape` is not validated during decode. > > **Overview** > Extends **`SamplingMask.from_sampling_mask`** so verifiers can ingest prime-rl’s packed CSR sampling masks from `/generate`, not only vLLM’s per-token `list[list[int]]` form. When the input is a dict with **`ids`** and **`counts`**, each value is decoded via a new **`_decode_packed`** helper (base64 `data` + `dtype` → `numpy` int32 arrays) instead of concatenating Python lists per token. > > List-based input is unchanged. A unit test asserts packed and list representations yield the same **`ids`** and **`counts`** arrays. > > Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 8f1a6ce. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).     > [!NOTE] > ### Accept packed CSR sampling masks in `SamplingMask.from_sampling_mask` > - `SamplingMask.from_sampling_mask` in [types.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2778/files#diff-88a4e4ee0c58ac5dc7344161a011374082a3767eda2ad32af97c942dca982112) now accepts either the existing list-of-lists input or a dict with packed `ids` and `counts` arrays, matching prime-rl's `/generate` format > - Adds a `_decode_packed` helper that base64-decodes the packed `data`, interprets it with the packed dtype, and casts to the requested dtype > - Adds a test in [test_trace.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2778/files#diff-ad6d1dda2b40e73baa4de5ad46c7a5a5580402aef93562589170bfc1bf9d1a99) verifying that list and packed inputs produce identical `ids` and `counts` arrays > - Risk: the packed `shape` metadata in `_decode_packed` is ignored; the arrays are reconstructed only from `counts` and `ids` > >  > > <a href="https://app.macroscope.com">Macroscope</a> summarized 8f1a6ce. >  >

Router replay now works with disaggregated P/D on vLLM 0.31, together with sampling replay. It replaces main's config rejection of router replay + P/D. **Problem on 0.31.** vLLM 0.31 captures routed experts through the AuxOutput connector (V2 runner only, vllm#45635) and rejects P/D connectors (`_verify_aux_output_compatibility`). The rejection is real: a decode (D) instance never computes a remotely prefilled prompt, so its worker would try to materialize the prompt rows from its block store (`BlockObjectStoreError`), and D's first forward (token `prompt_len-1`) starts mid-block, which the routed-experts buffer rejects. **Fix (plugin patch `monkey_patch_routed_experts_with_pd_connectors`, our upstream AuxOutput patch 02):** - Config: skip only the P/D connector check when the KV transfer config has `NixlConnector` (alone or inside `MultiConnector`). Every other AuxOutput check still runs. - Scheduler: a request on D (`kv_transfer_params` with a `remote_engine_id` and no `do_remote_decode`; NIXL clears `do_remote_prefill` but keeps that) emits rows only from its first forward on D (`num_computed_tokens` at that point: normally `prompt_len-1`, lower after a failed KV load, where D recomputes those rows itself). The floor survives preemption and is dropped when the request finishes. - Worker: the partially known first block is not buffered or published; later blocks D computes in full are published as usual. **Router v0.2.3 required (router#61); the decode `start` comes from #3900 (merged).** On a decode instance the rows start at its first forward, so the response carries only the rows D computed. #3900's packed `/generate` anchors the rows at the end: `start = len(prompt) + len(completion) - 1 - len(rows)` (equal to `routed_experts_prompt_start` without P/D). This PR does not touch `routed_experts.py`/`serving_tokens.py`, so it and #3900 (and #3920/#3794 on it) land in either order; P/D router replay needs both. Router v0.2.3 cuts decode rows by position (`(prefill.start + prefill.seq_len) - decode.start`) and also handles uint16 routing (models with more than 256 experts). This PR pins `vllm-router` to v0.2.3 in `pyproject.toml`/`uv.lock`. Without #3900, the merge would drop real completion rows. Also: removes `validate_disaggregated_no_routed_experts` (both validators on main), docs. Stack: **this PR** → #3831 (router replay with KV offload) → #3832 (P/D per-role KV offload) → #4023 (draft, HiSparse on decode). ## Checks run - `pytest -m "not gpu" tests/unit/test_configs.py tests/unit/inference` on this head: 214 passed. `ruff check` / `ruff format --check`: clean. - New tests (`test_routed_experts_pd.py`): the scheduler floor (first forward, preemption, cleanup) on vLLM's real `AuxOutputSchedulerConnector`, and the worker skip on the real `RoutedExpertsBuffer`. `test_combined_replay_uses_v2_runner` is parametrized over single-node and P/D. - vLLM's own `tests/distributed/aux_output_connector/test_store.py` (v0.31.0) with this patch applied plus patch 02's two new tests: 66 passed. Without the patch the two new tests fail. ## GPU validation (2026-10-06, vLLM 0.31.0, 1x 8xH200, Qwen3-30B-A3B TP1, `VLLM_BATCH_INVARIANT=1`, router v0.2.2 with zero-padded decode rows) - 1P1D (NIXL) vs colocated, router replay + sampling masks: 10/10 identical at concurrency 1 (prompts 64-6000 tokens, 128 completion tokens) and 20/20 at concurrency 10 with a fresh `cache_salt`. Identical means tokens, prompt and completion routing rows (100%) and sampling masks; rows = prompt + completion - 1. - Forced decode preemption (D with 1300 blocks, 20 requests, 1000-4000 prompts, up to 1024 tokens, concurrency 10): 6 preemptions on D, 20/20 identical to colocated. - All engines log `Using V2 Model Runner`; no `BlockObjectStoreError`. Re-run (job 2900, 2026-10-06, same setup, launched from per-role configs written by #3832's `write_pd_configs`): 1P1D (NIXL, no offload) vs colocated, router replay + masks: 10/10 identical at concurrency 1 and 20/20 at concurrency 10 with a fresh `cache_salt`; 0 errors. Not run: injected NIXL load failure (`kv_load_failure_policy=recompute`), MTP on decode, an `rl` run with a P/D config. ## GPU validation with router v0.2.3 (2026-10-14) 1x 8xH200, vLLM 0.31.0, Qwen3-30B-A3B TP1, `VLLM_BATCH_INVARIANT=1`, router replay + sampling masks. prime-rl = #3830 (978146c) merged with #3900 (20a3225); router = `release/v0.2.3` (ff818da, this PR + version bump), wheel built locally. 1P1D (NIXL) through the router vs a colocated server. Identical means: same tokens, same routing rows for prompt and completion (100%, rows = prompt + completion - 1, merged `start` = 0), same sampling masks, same logprobs. - Concurrency 1 (prompts 64-6000 tokens, 128 completion tokens): 10/10 identical. - Concurrency 10, fresh `cache_salt` on the colocated side: 20/20 identical. - uint16: no knob exists (vLLM picks uint8 for <=256 experts), so a test-only `sitecustomize` forced vLLM's capturer to `uint16` on P, D and the colocated server. The responses carry `dtype: uint16`. Concurrency 1: 10/10 identical; concurrency 10: 20/20 identical. - Forced decode preemption (D with `num_gpu_blocks_override=1300`, 1000-4000 token prompts, up to 1024 tokens, concurrency 10): 6 preemptions on D, 20/20 identical. - Router logs: no errors or warnings during the runs (only the shutdown warning). Engines: all `Using V2 Model Runner`, no `BlockObjectStoreError`. - Negative control, router v0.2.2 with the same P/D pair: 8/10 requests fail (HTTP 500 `decode routed_experts has 128 rows, expected at least <prompt_len>`). The 2 prompts of 64 tokens return 128 rows instead of 191, and the completion rows are wrong.  --- > [!NOTE] > **High Risk** > Changes expert-routing capture on NIXL P/D via vLLM monkey patches and removes prior safety guards; training correctness depends on router 0.2.3 merge behavior and patch edge cases (preemption, failed KV load). > > **Overview** > **Router replay** and **sampling replay** are now allowed on **disaggregated prefill/decode (P/D)** deployments, where they were previously blocked by config validation and vLLM’s AuxOutput + NIXL incompatibility. > > Config validators that rejected `enable_return_routed_experts` with `deployment.type = "disaggregated"` are removed from inference and RL configs. Docs now describe P/D behavior: prompt routing from prefill, completion routing from decode, and that sampling capture can run alongside router replay under P/D. > > A startup **vLLM monkey patch** (`monkey_patch_routed_experts_with_pd_connectors`) bypasses only the P/D connector check in `_verify_aux_output_compatibility`, raises an emit floor on decode so routed-expert rows start at the instance’s first forward, and skips buffering a partial first block after remote prefill. **`vllm-router` is pinned to v0.2.3** so the router can merge prefill/decode expert rows correctly. > > Unit tests cover scheduler floors, worker buffering, and V2 runner setup for both single-node and disaggregated configs. > > Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit e586ef8. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).  --------- Signed-off-by: faresobeid <111092724+faresobeid@users.noreply.github.com>

…nvs to latest main verifiers stays at main's pin (e888a719: v0.4.0 + the packed CSR sampling-mask decode that #3900's /generate payloads require) and the pyproject floor moves to the released verifiers[harbor]>=0.4.0; no new verifiers commits come in, so the unreleased refactors past main (task workspace deployments #2833, agentic judging in the workspace #2832, verification into the rollout lifecycle #2830, Runtime view split #2827) stay out of this bump. - renderers ca029b5 -> 5d870d8: GLM-5.3 Flash renderer with image support (#191); floor to the released renderers[multimodal]>=0.1.12 - prime-envs 89dcc7f -> 91d7d15: HLE variants + ALE CLI adapter (#979), verifiers>=0.4.0 required across envs (#983), lean memory (#986), nl2repobench offline VM images (#985), lab Prime image namespace (#984), terminal-bench-2 repaired QEMU images (#974), swe staged task inputs for Harbor 0.24 (#975), pi-terminal-bench-2 removed (#973) ale_cli joins the workspace excludes: it needs numpy>=2.4.4 (agents-last-exam tasks) while vLLM 0.31.0 caps numpy<2.4 via mistral-common. Validated: uv lock resolves; CPU unit tests pass (827 passed, 1 skipped, 236 deselected).

…nvs to latest main (#4056) Bumps the **renderers** and **prime-envs** submodules to latest main and moves the first-party pins to their released floors (**verifiers 0.4.0**, **renderers 0.1.12**). Re-landed on current main: the previous version of this PR conflicted with the v0.10.0 release. | dep | from | to | |---|---|---| | verifiers | `e888a71` (main's pin) | unchanged; pyproject floor -> released `>=0.4.0` | | renderers | `ca029b5` | `5d870d8` (main) | | prime-envs | `89dcc7f` | `91d7d15` (main) | **Why verifiers stays at main's pin instead of the `v0.4.0` tag:** main's pin is v0.4.0+12: it carries [#2778](PrimeIntellect-ai/verifiers#2778) (packed CSR sampling-mask decoding), which the packed per-token `/generate` payloads from #3900 require. Pinning the tag exactly would break router-replay runs: the tag's `SamplingMask.from_sampling_mask` only accepts the one-list-per-token form, while `/generate` emits packed `{ids, counts}`. So this PR holds verifiers where main has it (the 0.4.0 line) and only raises the floor to the released `0.4.0` — no new verifiers commits; the unreleased refactors past main (task workspace deployments #2833, agentic judging in the selected workspace #2832, verification into the rollout lifecycle #2830, Runtime view split from provider backends #2827) stay out. What comes in: - **renderers** (1 commit): GLM-5.3 Flash renderer with image support (#191) - **prime-envs** (9 commits): HLE variants and ALE CLI evaluation adapter (#979), require verifiers 0.4.0 across all environments (#983), fix lean memory for isolated proof grading (#986), fix nl2repobench offline VM image selection (#985), fix lab Prime image namespace (#984), fix terminal-bench-2 repaired QEMU task images (#974), fix swe staged task inputs for Harbor 0.24 (#975), remove pi-terminal-bench-2 (#973), speed up pull request smoke tests (#976) **Workspace change:** `deps/prime-envs/environments/office/ale_cli` is excluded from the workspace. It requires `numpy>=2.4.4` (via the agents-last-exam `tasks` package), and vLLM 0.31.0 requires `numpy<2.4` via `mistral-common`, so it can't share the lock (same treatment as tau2/apex). The lock changes only first-party packages: env versions move (deep-swe 0.1.2, hle 0.3.1, swe-sweep 0.1.1, terminal-bench-2 0.1.1; pi-terminal-bench-2 removed) and envs align on `verifiers>=0.4.0` with explicit `datasets>=3.0.0`. No third-party version moved. Validated locally: `uv lock` resolves and `uv lock --check` is clean; CPU unit tests pass against the bumped sources (`uv run pytest tests/unit -m "not gpu"`: **827 passed, 1 skipped, 236 deselected**). Not yet run: GPU/integration CI and a sandboxed RL smoke run.  --- > [!NOTE] > **Medium Risk** > Wide verifiers 0.4 rollout and workspace exclusion affect every env in the shared lock; upstream changes to verification and task workspaces may change RL/eval behavior even though prime-rl source is mostly unchanged. > > **Overview** > Bumps first-party training stack deps to released floors: **`verifiers[harbor]>=0.4.0`** and **`renderers[multimodal]>=0.1.12`** (dropping dev pre-release pins), with **`uv.lock`** refreshed so workspace env packages match. > > **Workspace:** **`deps/prime-envs/environments/office/ale_cli`** is added to **`tool.uv.workspace` `exclude`** because its numpy floor conflicts with vLLM’s numpy cap—same pattern as tau/apex envs. > > **Lock alignment:** Editable prime-envs now require **`verifiers>=0.4.0`**; many dataset-driven envs also declare explicit **`datasets>=3.0.0`**. **`pi-terminal-bench-2`** drops out of the workspace/lock; a few env package versions tick up (e.g. **deep-swe**, **hle**, **swe-sweep**, **terminal-bench-2**). No third-party PyPI version churn beyond first-party metadata in the lock. > > Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit b9750ab. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).

This was referenced Oct 6, 2026

perf(client): read prime-rl's packed completion_logprobs PrimeIntellect-ai/renderers#174

Merged

perf(v1): accept packed CSR sampling masks from prime-rl's /generate PrimeIntellect-ai/verifiers#2778

Merged

Base automatically changed from chore/bump-vllm-0.31 to main October 6, 2026 14:46

faresobeid force-pushed the perf/packed-generate-gc branch from 2f80573 to 355d9b2 Compare October 6, 2026 14:53

This was referenced Oct 6, 2026

chore: bump verifiers (race-free uv script cache) #3897

Closed

feat(router): configurable vllm-router health check and circuit breaker #3895

Closed

faresobeid marked this pull request as ready for review October 6, 2026 17:22

cursor Bot reviewed Oct 6, 2026

View reviewed changes

Comment thread src/prime_rl/inference/patches.py

faresobeid force-pushed the perf/packed-generate-gc branch from 355d9b2 to df49fe2 Compare October 7, 2026 00:39

cursor Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread src/prime_rl/inference/patches.py

faresobeid mentioned this pull request Oct 7, 2026

fix(inference): cut router replay's engine-side cost on vLLM 0.31 #3920

Open

faresobeid force-pushed the perf/packed-generate-gc branch from 8760073 to 601f185 Compare October 8, 2026 14:35

faresobeid mentioned this pull request Oct 8, 2026

feat: exact score centering over the sampling mask (arXiv:2609.20807) #3794

Open

cursor Bot reviewed Oct 8, 2026

View reviewed changes

Comment thread src/prime_rl/inference/patches.py

faresobeid mentioned this pull request Oct 8, 2026

feat(inference): default api_server_count to 4 for single-engine servers #3943

Closed

faresobeid force-pushed the perf/packed-generate-gc branch 2 times, most recently from c5f8b5a to bb77d68 Compare October 9, 2026 23:48

cursor Bot reviewed Oct 9, 2026

View reviewed changes

Comment thread src/prime_rl/inference/patches.py

faresobeid mentioned this pull request Oct 10, 2026

feat: store by-handle sampling masks as CSR #3962

Closed

faresobeid added 2 commits October 10, 2026 15:13

faresobeid force-pushed the perf/packed-generate-gc branch from bb77d68 to b920328 Compare October 10, 2026 15:21

faresobeid added 2 commits October 12, 2026 18:27

Merge remote-tracking branch 'origin/main' into perf/packed-generate-gc

4e3cf50

refactor(inference): serialize routed experts with encode_array

576f771

refactor(inference): anchor packed routed-experts start at the last f…

20a3225

…orwarded token start = prompt + completion - 1 - rows. Equals routed_experts_prompt_start on a colocated instance; on a P/D decode instance the rows begin at its first forward.

faresobeid added a commit that referenced this pull request Oct 13, 2026

refactor(inference): take the end-anchored routing start from the pac…

978146c

…ked /generate path The start moves into _PackedOutputs (#3900), so routed_experts.py and serving_tokens.py stay as on main.

This was referenced Oct 13, 2026

feat(inference): router replay on the V2 runner everywhere #3830

Merged

fix(pd): honor routed_experts dtype in inline merge PrimeIntellect-ai/router#61

Merged

chore(deps): renderers main (#174 merged), verifiers #2778 rebased on…

7ac3b9f

… main

chore(deps): verifiers main (#2778 merged)

a6f0da5

mikasenghaas approved these changes Oct 14, 2026

View reviewed changes

faresobeid merged commit aa2f994 into main Oct 14, 2026
16 of 20 checks passed

faresobeid deleted the perf/packed-generate-gc branch October 14, 2026 15:44

samsja mentioned this pull request Oct 14, 2026

chore(deps): verifiers floor to released 0.4.0; renderers and prime-envs to latest main #4056

Merged

Navigation Menu

Repository navigation

perf(inference): pack /generate per-token payloads and tune API-server GC - #3900

perf(inference): pack /generate per-token payloads and tune API-server GC#3900
faresobeid merged 7 commits into
mainfrom
perf/packed-generate-gc

faresobeid commented Oct 6, 2026 •

edited by cursor Bot

Loading

Uh oh!

cursor Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

cursor Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

cursor Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

faresobeid commented Oct 6, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Change

Pins

Measurement

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

faresobeid commented Oct 6, 2026 •

edited by cursor Bot

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading