Skip to content

SL-51 PR1: wire 6 VLM-registry overlays; text-only models share a processor shim - #107

Merged
solderzzc merged 1 commit into
mainfrom
sl51/pr1-text-shim
Nov 2, 2026
Merged

solderzzc merged 1 commit into
mainfrom
sl51/pr1-text-shim

Conversation

@solderzzc

Copy link
Copy Markdown
Member

What

SL-51, first slice. Wires 6 of the 71 VLM overlay implementations into VLMModelFactory, and adds a processor shim so models without a ported vision or audio tower can load through the VLM factory.

  • Shim: SharpAITextOnlyProcessor renders the chat template and tokenizes. It throws mediaNotSupported for images, video or audio instead of dropping them. SharpAITextOnlyProcessorLoadingResolver forces this processor for the text-only model_types whatever processor_class a checkpoint declares, and supplies an empty configuration when preprocessor_config.json is absent. It plugs into the existing VLMProcessorLoadingRegistry; no shared loading code changed.
  • Wired keys (SharpAIVLMModelOverlays.vlmWiredKeys): interns1_pro, interns2_mobius, kimi_k3, qwen4_exp, step1 (text-only, use the shim) and intern_s2_preview (subclasses upstream Qwen3.5, keeps its own processor).
  • Policy: upstream wins, same as SL-50. The merge is two lines in VLMModelFactory.swift plus one resolver entry in VLMProcessorLoadingRegistry.shared.
  • Left out on purpose: the 29 model_types whose creator always throws or returns a model that cannot run. A test pins that none of them is registered.
  • Conformance tooling: gen_expectations.py and missing-ops/gen_report.py read the new VLM aggregate; expectations and report regenerated (VLM registry keys 22 → 28).
  • No change to .github/workflows or Package.swift.

Tests

  • SharpAIVLMOverlaysTests: wired keys resolve in VLMTypeRegistry.shared; aggregate holds exactly them; 29 blocked keys stay unregistered; resolver forces the shim for the 5 text-only types and leaves other models alone; shim tokenizes text with no mask and rejects media.
  • SharpAIVLMAirgapLoadTests: for each of the 6 keys, a deterministic non-zero synthetic checkpoint (config.json + safetensors, generated locally, nothing downloaded) loads through the real loader and reproduces the logits via prepare. coversEveryWiredKey fails if a wired key has no case.
  • Also run locally: SharpAIModelOverlaysTests, SharpAIAirgapLoadTests, ConformanceRegistryTests, ConformanceMissingOpsTests, ConformanceSwigluGoldenTests, VLMProcessorLoadingRegistryTests, VLMRegistryTests, VLMTextOnlyContinuationTests. Test process EXIT=0.

Not covered

  • The air-gap test checks the model creator and loader path. It does not run VLMModelFactory._load end to end (that needs a tokenizer loader), and the synthetic file uses the model's own parameter names, so it does not prove the sanitizers match a real checkpoint.
  • None of the 6 has been loaded from a real checkpoint. Real processor_class values for these models are unverified; the resolver overrides them on purpose.

Scouting report and the plan for the remaining PRs: docs/research/sl51-vlm-scout-2026-11-01.md (kept out of the repo).

🤖 Generated with Claude Code

…cessor shim

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@solderzzc
solderzzc merged commit 188b384 into main Nov 2, 2026
11 of 17 checks passed
@solderzzc
solderzzc deleted the sl51/pr1-text-shim branch November 4, 2026 05:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant