Repository navigation
SL-51 PR1: wire 6 VLM-registry overlays; text-only models share a processor shim - #107
Merged
Merged
Conversation
…cessor shim Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Nov 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
SL-51, first slice. Wires 6 of the 71 VLM overlay implementations into
VLMModelFactory, and adds a processor shim so models without a ported vision or audio tower can load through the VLM factory.SharpAITextOnlyProcessorrenders the chat template and tokenizes. It throwsmediaNotSupportedfor images, video or audio instead of dropping them.SharpAITextOnlyProcessorLoadingResolverforces this processor for the text-onlymodel_types whateverprocessor_classa checkpoint declares, and supplies an empty configuration whenpreprocessor_config.jsonis absent. It plugs into the existingVLMProcessorLoadingRegistry; no shared loading code changed.SharpAIVLMModelOverlays.vlmWiredKeys):interns1_pro,interns2_mobius,kimi_k3,qwen4_exp,step1(text-only, use the shim) andintern_s2_preview(subclasses upstream Qwen3.5, keeps its own processor).VLMModelFactory.swiftplus one resolver entry inVLMProcessorLoadingRegistry.shared.model_types whose creator always throws or returns a model that cannot run. A test pins that none of them is registered.gen_expectations.pyandmissing-ops/gen_report.pyread the new VLM aggregate; expectations and report regenerated (VLM registry keys 22 → 28)..github/workflowsorPackage.swift.Tests
SharpAIVLMOverlaysTests: wired keys resolve inVLMTypeRegistry.shared; aggregate holds exactly them; 29 blocked keys stay unregistered; resolver forces the shim for the 5 text-only types and leaves other models alone; shim tokenizes text with no mask and rejects media.SharpAIVLMAirgapLoadTests: for each of the 6 keys, a deterministic non-zero synthetic checkpoint (config.json + safetensors, generated locally, nothing downloaded) loads through the real loader and reproduces the logits viaprepare.coversEveryWiredKeyfails if a wired key has no case.EXIT=0.Not covered
VLMModelFactory._loadend to end (that needs a tokenizer loader), and the synthetic file uses the model's own parameter names, so it does not prove the sanitizers match a real checkpoint.processor_classvalues for these models are unverified; the resolver overrides them on purpose.Scouting report and the plan for the remaining PRs:
docs/research/sl51-vlm-scout-2026-11-01.md(kept out of the repo).🤖 Generated with Claude Code