From dd2755aa4548370679c90c914c30c045d6575dad Mon Sep 17 00:00:00 2001 From: CJ Pais Date: Sat, 11 Jul 2026 07:50:41 -0700 Subject: [PATCH 01/13] stage 1: --- docs/porting/families/moss.md | 110 ++ .../_research_dump.txt | 1152 +++++++++++++++++ .../moss/moss-transcribe-diarize/intake.json | 164 +++ .../moss-transcribe-diarize.manifest.json | 12 + 4 files changed, 1438 insertions(+) create mode 100644 docs/porting/families/moss.md create mode 100644 reports/porting/moss/moss-transcribe-diarize/_research_dump.txt create mode 100644 reports/porting/moss/moss-transcribe-diarize/intake.json create mode 100644 tests/golden/moss/moss-transcribe-diarize.manifest.json diff --git a/docs/porting/families/moss.md b/docs/porting/families/moss.md new file mode 100644 index 00000000..bfced132 --- /dev/null +++ b/docs/porting/families/moss.md @@ -0,0 +1,110 @@ +# MOSS-Transcribe-Diarize + +Status: research + +## Identity + +- Family key: `moss` +- Upstream architecture string: `MossTranscribeDiarizeForConditionalGeneration` (`model_type: moss_transcribe_diarize`) +- Hugging Face repo: `OpenMOSS-Team/MOSS-Transcribe-Diarize` +- Hugging Face revision: `d7231bbae2587a4af278735eb765b318c4f64edd` +- License: Apache-2.0 +- Variants: `moss-transcribe-diarize` (0.9B; Whisper-Medium encoder + VQAdaptor + Qwen3-0.6B decoder) + +Architecture pattern: **audio-llm**. log-mel `input_features` → `WhisperEncoder` +(24 layers, d_model 1024, gelu, LayerNorm) → 4× temporal merge (1024 → 4096) → +`VQAdaptor` MLP (Linear 4096→1024, SiLU, Linear 1024→1024, LayerNorm w/ bias) → +`masked_scatter` into Qwen3-0.6B text embeddings at `<|audio_pad|>` (id 151671) +positions → Qwen3 decoder (28 layers, hidden 1024, silu, RMSNorm, GQA 16/8 heads, +head_dim 128, rope_theta 1e6, tie_word_embeddings) → `lm_head`. + +Output is emergent plain text in the canonical format `[start][Sxx]text[end]` +(e.g. `[0.48][S01]Welcome[1.66]`); diarization speaker tags and segment +timestamps are generated text, not special tokens. + +## References + +- Canonical reference: `author_repo_moss` — OpenMOSS custom `trust_remote_code` + files in the HF repo (`modeling_moss_transcribe_diarize.py`, + `configuration_moss_transcribe_diarize.py`, + `processing_moss_transcribe_diarize.py`) plus the OpenMOSS/MOSS-Transcribe-Diarize + GitHub inference harness (`build_transcription_messages`, `generate_transcription`, + audio/video loading, transcript parsing). Not in upstream Transformers. +- Instrumented reference: same custom code (single source of truth; no framework fork). +- Cross-check references: `Qwen3Model` and `WhisperEncoder` upstream Transformers + implementations that the custom code composes. + +## Commands + +Reference run: + +```bash +TODO # porting-2-oracle: transformers + trust_remote_code=True generate() over samples/jfk.wav +``` + +Reference dumps: + +```bash +TODO # porting-2-oracle +``` + +Conversion: + +```bash +TODO # porting-3-convert +``` + +Validation: + +```bash +TODO # porting-4-cpp +``` + +Benchmarks: + +```bash +TODO # porting-6-bench +``` + +## Capability Validation + +| Capability | Mode | Command / test | Expected observable | Target | Status | +|------------|------|----------------|---------------------|--------|--------| +| Transcribe | auto / no language hint (en) | `build/bin/transcribe-cli -m models/moss-transcribe-diarize/moss-transcribe-diarize-.gguf samples/jfk.wav` | non-empty plausible English transcript | MUST PASS | TODO | +| Transcribe | auto / no language hint (zh) | `build/bin/transcribe-cli -m models/moss-transcribe-diarize/moss-transcribe-diarize-.gguf ` | non-empty plausible Chinese transcript | MUST PASS | TODO | +| Transcribe | explicit language hint | `build/bin/transcribe-cli -m ... --language en samples/jfk.wav` | non-empty plausible transcript | OUT OF SCOPE — no first-class language token; language is inferred from audio and steered only by prompt text. Back in scope if the runtime exposes a prompt/language selector. | TODO | +| Speaker diarization | default diarize prompt | `build/bin/transcribe-cli -m ... ` | `[Sxx]` speaker tags present in the raw transcript | OUT OF SCOPE — diarized `[Sxx]` output is produced correctly in the raw transcript (rides the same output format the Segment-timestamps row gates), but there is no runtime API surface to expose diarization as a distinct capability; a dedicated API surface is out of scope for this PR. Back in scope when that API lands. | TODO | +| Segment timestamps | default diarize prompt | `build/bin/transcribe-cli -m ... samples/jfk.wav` | output contains `[start]...[end]` numeric timestamps | MUST PASS | TODO | +| Word timestamps | — | — | word-granularity timestamps | OUT OF SCOPE — model emits segment-level `[start][end]` only; no word alignment. | TODO | +| Long-form audio | >30s multi-chunk | `build/bin/transcribe-cli -m ...