test: freeze renderer behavior before refactoring

.github/workflows/test.yml

@@ -5,37 +5,99 @@
     branches: [main]
     paths:
       - "**.py"
+      - "tests/golden_renderer_outputs.json"
       - "pyproject.toml"
       - ".github/workflows/test.yml"
   pull_request:
     branches: [main]
     paths:
       - "**.py"
+      - "tests/golden_renderer_outputs.json"
       - "pyproject.toml"
       - ".github/workflows/test.yml"
 
 jobs:
-  test:
-    name: Renderers
+  unit:
+    name: Offline unit tests (Python ${{ matrix.python-version }})
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
       matrix:
         python-version: ["3.10", "3.11", "3.12", "3.13"]
 
     steps:
       - uses: actions/checkout@v4
 
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v6
         with:
           python-version: ${{ matrix.python-version }}
 
       - name: Install uv
         uses: astral-sh/setup-uv@v7
 
       - name: Install dependencies
         run: uv sync
 
       - name: Run tests
-        run: uv run pytest tests/ -v
+        run: uv run pytest tests/ -m "not network" -v
+
+  model-parity:
+    name: Pinned model parity
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    env:
+      HF_HUB_DISABLE_PROGRESS_BARS: "1"
+      RENDERERS_TEST_NETWORK: "1"
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python 3.13
+        uses: actions/setup-python@v6
+        with:
+          python-version: "3.13"
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7
+
+      - name: Cache pinned Hugging Face assets
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/huggingface
+          key: hf-renderers-${{ hashFiles('tests/model_assets.py') }}
+
+      - name: Install dependencies
+        run: uv sync
+
+      - name: Run text model parity tests
+        run: uv run pytest tests/ -m "network and not multimodal" -v
+
+  multimodal-parity:
+    name: Pinned multimodal parity
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    env:
+      HF_HUB_DISABLE_PROGRESS_BARS: "1"
+      RENDERERS_TEST_NETWORK: "1"
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python 3.13
+        uses: actions/setup-python@v6
+        with:
+          python-version: "3.13"
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7
+
+      - name: Cache pinned Hugging Face assets
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/huggingface
+          key: hf-renderers-${{ hashFiles('tests/model_assets.py') }}
+
+      - name: Install dependencies
+        run: uv sync
+
+      - name: Run multimodal parity tests
+        run: uv run pytest tests/ -m "multimodal" -v

README.md

@@ -33,14 +33,19 @@ parsed = r.parse_response(completion_ids)
 For the next turn, extend the previous sampled stream instead of re-rendering history:
 
 ```python
-next_prompt_ids = r.bridge_to_next_turn(
+bridged = r.bridge_to_next_turn(
     previous_prompt_ids=prompt_ids,
     previous_completion_ids=completion_ids,
     new_messages=[{"role": "tool", "content": "..."}],
 )
+if bridged is None:
+    # The renderer could not prove that preserving the sampled prefix is safe.
+    # Fall back to a full render of the conversation in that case.
+    ...
+next_prompt_ids = bridged.token_ids
 ```
 
-Hand-coded renderers ship for `qwen3`, `qwen3-vl`, `qwen3.5`, `qwen3.6`, `glm-5`, `glm-5.1`, `glm-4.5`, `minimax-m2`, `deepseek-v3`, `deepseek-r1`, `kimi-k2`, `kimi-k2.5` / `kimi-k2.6`, `nemotron-3`, `nemotron-3-ultra`, `llama-3`, `gpt-oss`, `hy3`, and `prime-qwen3`. Anything else falls back to `DefaultRenderer`, a generic `apply_chat_template` wrapper.
+Hand-coded renderers ship for `qwen3`, `qwen3-vl`, `qwen3.5`, `qwen3.6`, `glm-5`, `glm-5.1`, `glm-4.5`, `minimax-m2`, `deepseek-v3`, `deepseek-r1`, `kimi-k2`, `kimi-k2.5` / `kimi-k2.6`, `laguna-xs.2`, `laguna-xs-2.1`, `nemotron-3`, `nemotron-3-ultra`, `llama-3`, `gpt-oss`, `hy3`, and `prime-qwen3`. Anything else falls back to `DefaultRenderer`, a generic `apply_chat_template` wrapper.
 
 ## API
 
@@ -50,7 +55,7 @@ class Renderer(Protocol):
     def render_ids(messages, *, tools=None, add_generation_prompt=False) -> list[int]: ...
     def parse_response(token_ids) -> ParsedResponse: ...
     def get_stop_token_ids() -> list[int]: ...
-    def bridge_to_next_turn(prev_prompt_ids, prev_completion_ids, new_messages, *, tools=None) -> list[int] | None: ...
+    def bridge_to_next_turn(prev_prompt_ids, prev_completion_ids, new_messages, *, tools=None) -> RenderedTokens | None: ...
 ```
 
 - `RenderedTokens` carries `token_ids` **and** `message_indices` — one entry per token attributing each to its source message (`-1` for structural scaffolding). Lets `build_training_sample` build a per-token loss mask in one render.
@@ -59,7 +64,7 @@ class Renderer(Protocol):
 
 ### `bridge_to_next_turn` (the core contract)
 
-Given `(prev_prompt_ids, prev_completion_ids)` and new environment messages, return ids for the next turn's prompt such that the result starts with `prev_prompt_ids + prev_completion_ids` byte-for-byte and continues with the new messages plus the next assistant opener. If that cannot be proven safe, return `None` and the caller falls back to a full render.
+Given `(prev_prompt_ids, prev_completion_ids)` and new environment messages, return a `RenderedTokens` object for the next turn's prompt whose `token_ids` start with `prev_prompt_ids + prev_completion_ids` byte-for-byte and continue with the new messages plus the next assistant opener. If that cannot be proven safe, return `None` and the caller falls back to a full render. Attribution in a bridge result is relative to `new_messages`; the preserved prefix uses `message_indices=-1` because only its raw token IDs are available.
 
 Each hand-coded bridge:
 1. Anchors at the previous turn's canonical close token. On clean stops it's already in `prev_completion_ids`. On truncation, the renderer synthesizes the close as non-loss prompt context.

examples/README.md

@@ -86,5 +86,9 @@ Each script runs `Qwen/Qwen3.5-4B` with `enable_thinking=True` and `False`, then
 
 ## Multimodal Note
 
-Renderers are text-only today. For image/video demos, use the backend's message
-or prompt path until renderers grow multimodal placeholder support.
+Image rendering is supported for Qwen3-VL, Qwen3.5 / Qwen3.6, and Kimi K2.5 /
+K2.6. Their renderers return token IDs plus a framework-agnostic
+`multi_modal_data` sidecar containing placeholder ranges and processed image
+features. `renderers.client` can serialize the Qwen-VL family sidecar for
+vLLM's token-in endpoint; other backends need their own adapter. Video content
+parts are represented by the public types but are not yet rendered.

pyproject.toml

@@ -117,3 +117,10 @@ invalid-key = "warn"
 unsupported-operator = "warn"
 invalid-return-type = "warn"
 invalid-assignment = "warn"
+
+[tool.pytest.ini_options]
+markers = [
+    "network: requires pinned external model assets from Hugging Face",
+    "model_parity: compares a renderer with a pinned upstream tokenizer or reference encoder",
+    "multimodal: exercises image/video processors and multimodal sidecars",
+]

renderers/base.py

@@ -1190,11 +1190,22 @@ def _tokenizer_source_for(model_name_or_path: str) -> str:
     return TOKENIZER_SOURCE_OVERRIDES.get(model_name_or_path, model_name_or_path)
 
 
-def _tokenizer_load_kwargs(model_name_or_path: str) -> dict[str, Any]:
-    revision = TRUSTED_REVISIONS.get(model_name_or_path)
+def _tokenizer_load_kwargs(
+    model_name_or_path: str, *, revision: str | None = None
+) -> dict[str, Any]:
+    trusted_revision = TRUSTED_REVISIONS.get(model_name_or_path)
+    if trusted_revision is not None:
+        if revision is not None and revision != trusted_revision:
+            raise ValueError(
+                f"{model_name_or_path!r} executes trusted remote tokenizer code "
+                f"only at reviewed revision {trusted_revision}; received "
+                f"revision={revision!r}."
+            )
+        return {"trust_remote_code": True, "revision": trusted_revision}
+    kwargs: dict[str, Any] = {"trust_remote_code": False}
     if revision is not None:
-        return {"trust_remote_code": True, "revision": revision}
-    return {"trust_remote_code": False}
+        kwargs["revision"] = revision
+    return kwargs
 
 
 def _preserve_requested_tokenizer_name(
@@ -1281,13 +1292,16 @@ def _load_tokenizer_via_auto(model_name_or_path: str, **kwargs) -> Any:
         return tok
 
 
-def load_tokenizer(model_name_or_path: str):
+def load_tokenizer(model_name_or_path: str, *, revision: str | None = None):
     """Load a tokenizer with the renderers-package security policy.
 
     Default ``trust_remote_code=False``. Models listed in
     ``TRUSTED_REVISIONS`` (Moonshot Kimi-K2 family) load with
     ``trust_remote_code=True`` AND a pinned ``revision=<sha>`` so
     transformers only executes the reviewed commit's tokenizer Python.
+    Callers may pin ``revision`` for repositories that do not execute remote
+    code (for example reproducible parity tests). A caller-supplied revision
+    may not override a reviewed ``TRUSTED_REVISIONS`` entry.
 
     ``AutoTokenizer.from_pretrained`` eagerly builds the model config to
     resolve the tokenizer class. If that construction raises on a
@@ -1302,7 +1316,7 @@ def load_tokenizer(model_name_or_path: str):
     the requested Meta ID so auto-resolution still selects ``Llama3Renderer``.
     """
     load_name_or_path = _tokenizer_source_for(model_name_or_path)
-    kwargs = _tokenizer_load_kwargs(load_name_or_path)
+    kwargs = _tokenizer_load_kwargs(load_name_or_path, revision=revision)
     tok = _load_tokenizer_via_auto(load_name_or_path, **kwargs)
     return _preserve_requested_tokenizer_name(
         tok,

tests/__init__.py

@@ -0,0 +1 @@
+"""Test support package for renderers."""

tests/conftest.py

@@ -5,11 +5,62 @@
 """
 
 import os
+from pathlib import Path
 
 import pytest
 from renderers import create_renderer
-from renderers.base import load_tokenizer
 from renderers.configs import config_from_name
+from tests.model_assets import load_test_tokenizer
+
+
+# Tests in these modules exercise real, pinned model assets. Keeping the tier
+# declaration here avoids hundreds of repeated decorators on parameterized
+# cases while still making ``pytest -m 'not network'`` a complete offline
+# suite. Mixed unit/network modules use per-test decorators instead.
+_MODEL_ASSET_TEST_MODULES = frozenset(
+    {
+        "test_bridge",
+        "test_build_helpers",
+        "test_deepseek_r1",
+        "test_disabled_thinking_stability",
+        "test_glm_tool_name_validation",
+        "test_golden_renderer_outputs",
+        "test_gpt_oss_harmony_parity",
+        "test_hy3",
+        "test_is_content",
+        "test_laguna_xs21",
+        "test_llama_3",
+        "test_message_indices",
+        "test_message_tool_names",
+        "test_multimodal",
+        "test_nemotron3_parity",
+        "test_parse_response",
+        "test_parse_response_robustness",
+        "test_parsers",
+        "test_preserve_thinking",
+        "test_prime_qwen3_parity",
+        "test_qwen35_size_coverage",
+        "test_render_ids",
+        "test_renderer_config_parity",
+        "test_roundtrip",
+        "test_sampled_mask",
+        "test_tokens_per_message",
+        "test_tool_arg_type_preservation",
+    }
+)
+
+
+def pytest_collection_modifyitems(items):
+    """Assign network/parity tiers without changing test parametrization."""
+    for item in items:
+        module_name = Path(str(item.path)).stem
+        if module_name not in _MODEL_ASSET_TEST_MODULES:
+            continue
+        item.add_marker(pytest.mark.network)
+        item.add_marker(pytest.mark.model_parity)
+        if module_name == "test_multimodal":
+            item.add_marker(pytest.mark.multimodal)
+
 
 # (HuggingFace model name, renderer name or "auto")
 #
@@ -65,7 +116,7 @@
 def _load(model_name: str, renderer_name: str):
     key = f"{model_name}:{renderer_name}"
     if key not in _cache:
-        tokenizer = load_tokenizer(model_name)
+        tokenizer = load_test_tokenizer(model_name)
         renderer = create_renderer(tokenizer, config_from_name(renderer_name))
         _cache[key] = (tokenizer, renderer)
     return _cache[key]

tests/generate_renderer_goldens.py

@@ -0,0 +1,33 @@
+"""Regenerate the checked-in renderer behavior corpus.
+
+Run from the repository root:
+
+    uv run python -m tests.generate_renderer_goldens
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+from tests.golden_corpus import GOLDEN_CASES, build_golden_case
+
+
+OUTPUT_PATH = Path(__file__).with_name("golden_renderer_outputs.json")
+
+
+def main() -> None:
+    cases = {}
+    for case in GOLDEN_CASES:
+        print(f"rendering {case.slug} ({case.model_name})", flush=True)
+        cases[case.slug] = build_golden_case(case)
+    payload = {"schema_version": 1, "cases": cases}
+    OUTPUT_PATH.write_text(
+        json.dumps(payload, indent=2, ensure_ascii=False) + "\n",
+        encoding="utf-8",
+    )
+    print(f"wrote {OUTPUT_PATH}", flush=True)
+
+
+if __name__ == "__main__":
+    main()