💻 GitHub   |   🤗 Hugging Face   |   🤖 ModelScope   |   📑 Blog   |   📚 Paper
🚀 HF Space   |   💬 Feishu   |   🫨 Discord   |   📜 License

English | 中文

MOSS-VL

MOSS-VL is an open-weight model series for long-form, real-time video understanding, built on a unified cross-attention architecture. All three models are 11B-parameter and open-weight.

• MOSS-VL-Realtime: real-time interaction over continuous video streams — interruptible at any moment, answering on the fly, and deciding on its own when to respond and when to keep watching.
• MOSS-VL-Instruct: built for offline use, with particular strength in complex long-video understanding and in-depth dialogue.
• MOSS-VL-Base: an open pre-trained foundation offering strong video–language representations for continued pre-training and downstream fine-tuning.

Unlike the default paradigm of offline video models ("watch first, answer after"), MOSS-VL-Realtime is designed for real-time interaction on continuous video streams: it runs multimodal perception and text generation in parallel on a continuously arriving stream, natively supporting multi-turn real-time dialogue and dynamic scene understanding, autonomously deciding when to speak, achieving fine-grained temporal grounding, and streaming its responses.

---

Key Enhancements

• Interruptible & Real-time: Breaking the limits of offline processing, users can ask questions at any timestamp in the video stream. The model instantly responds based on the currently received frames, achieving open-source SOTA latency in streaming interaction.
• Proactive Silence: Equipped with dynamic judgment capabilities, the model autonomously stays silent and continues observing when contextual information is insufficient or no key events have occurred.
• Dynamic Correction: Cognition updates dynamically with the visual feed. As new frames continue to stream in, the model can instantly capture state changes and correct its previous outputs.

Core Architecture

At the architectural level, MOSS-VL-Realtime adopts the following core designs:

• Cross-Attention Architecture: Decouples visual encoding from language reasoning, significantly reducing response latency on dynamic video streams, with native support for interleaved image, video, and text inputs.
• Absolute Timestamps: Every sampled frame is anchored to a precise time marker via dedicated special tokens. The model uses this to reason about pacing, duration, and motion dynamics, and it natively adapts to variable frame rates.
• XRoPE (Cross-Attention Rotary Position Embedding): Maps text tokens and video patches into a unified 3D (t, h, w) coordinate space, enabling patch-level and moment-level grounding across the entire video.

---

🔥 News

• 2026/07/14: 🚀 Released MOSS-VL-Realtime for real-time video understanding on continuous streams, together with the new MOSS-VL-Instruct-0708 and MOSS-VL-Base-0708.
• 2026/04/24: 🚀 SGLang officially supports MOSS-VL; see sgl-project/sglang.
• 2026/04/22: 🚀 Released SGLang-based inference support for MOSS-VL. See ./sglang/.
• 2026/04/22: 🤗 Updated HuggingFace inference code to the latest version.
• 2026/04/08: 🚀 Released MOSS-VL-Base-0408 and MOSS-VL-Instruct-0408.

---

📊 Performance

MOSS-VL-Realtime delivers significantly stronger streaming interaction capabilities, achieving open-source SOTA results on multiple streaming video understanding benchmarks. Its "proactive speaking" ability stands out in particular: the model leads on all three proactivity evaluations — Proactive Alerting in OmniMMI, Proactive Output in StreamingBench, and ProactiveVideoQA.

Streaming Benchmark

We have systematically restructured and deeply optimized our data system, comprehensively strengthening the model's foundational capabilities and instruction-interaction experience, while maintaining a high level of stability across offline evaluations.

Offline Multimodal Benchmark — Click to Expand

For comprehensive benchmark breakdowns, comparison systems, and detailed tables of all objective metrics, please refer to our Technical Blog.

---

🚀 Quick Start

Environment Setup

conda create -n moss_vl python=3.12 pip -y
conda activate moss_vl
pip install -i https://pypi.org/simple --no-build-isolation -r requirements.txt

Real-time Inference

Real-time inference consumes timestamped frames incrementally, so the model can keep perceiving a live video stream while it answers and can accept new questions at any time. The fastest way to replay a local video against its media clock is:

CUDA_VISIBLE_DEVICES=0 python realtime_inference/run_online_inference.py \
  --checkpoint OpenMOSS-Team/MOSS-VL-Realtime \
  --source video \
  --video path/to/example.mp4 \
  --sample-fps 1 \
  --playback-speed 1 \
  --max-frames 256

Keep --playback-speed 1 for model inference so frames arrive on the original timeline. The runtime provides three integration levels:

• model.create_realtime_session(...) for direct frame, prompt, and output control
• model.online_generate(...) for queue-based inference workers
• --serve for a FastAPI WebSocket service that accepts external JPEG/PNG frames or replays server-local videos

It also supports streaming JSONL samples, cameras, screen capture, and synthetic sources. See realtime_inference/README.md for the complete CLI, input format, and WebSocket protocol.

Offline Inference

Offline inference supports full-modality queries (interleaved text, image, and video inputs). The fastest way to get a first result is offline_batch_generate:

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

checkpoint = "OpenMOSS-Team/MOSS-VL-Realtime"

processor = AutoProcessor.from_pretrained(checkpoint, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    checkpoint, trust_remote_code=True, device_map="auto", torch_dtype=torch.bfloat16
)

queries = [{
    "messages": [{"role": "user", "content": [
        {"type": "image", "image": "path/to/example.jpg"},
        {"type": "text", "text": "Describe this image."}
    ]}],
    "generate_kwargs": {"max_new_tokens": 256, "do_sample": False},
}]

with torch.no_grad():
    result = model.offline_batch_generate(processor, queries)

print([item["text"] for item in result["results"]])

---

🛠️ Advanced Resources & Ecosystem

Deployment & Inference Engines

MOSS-VL can also be efficiently deployed with the following inference backends:

• SGLang: see sglang/README.md

Fine-Tuning

We provide a lightweight SFT framework built on HuggingFace transformers.Trainer. It supports full-parameter training and LoRA, with the vision encoder, language model, and LM head independently controllable.

# Full-parameter SFT (vision encoder frozen by default)
bash mossvl_finetune/scripts/run_sft.sh

# LoRA SFT
pip install -i https://pypi.org/simple peft
bash mossvl_finetune/scripts/run_sft_lora.sh

See mossvl_finetune/README.md for full documentation.

Model Download

This generation ships three models from the same rebuilt data: MOSS-VL-Realtime for continuous video streams, Instruct for offline tasks, and Base for continued pre-training and fine-tuning.

Model	Params	Context	Best for	🤗 HuggingFace	🤖 ModelScope
MOSS-VL-Realtime	11B	256K	Real-time interaction on continuous video streams	Link	Link
MOSS-VL-Instruct-0708	11B	256K	Offline chat / inference / downstream tasks	Link	Link
MOSS-VL-Base-0708	11B	256K	Continued pre-training / fine-tuning	Link	Link

Previous generation:

Model	Params	Context	Best for	🤗 HuggingFace	🤖 ModelScope
MOSS-VL-Base-0408	11B	256K	Continued pre-training / fine-tuning	Link	Link
MOSS-VL-Instruct-0408	11B	256K	Chat / inference / downstream tasks	Link	Link

---

📑 Roadmap & TODO List

✅ Milestones

• Core Architecture: Implementation of Cross-attention RoPE (XRoPE).
• High-performance Infra: Integrated Megatron-LM + CUDA Flash Attention 3.
• Model Release: Open-sourced MOSS-VL-Base and MOSS-VL-Instruct.
• Inference: Inference code for both image and video understanding.
• Real-time Capabilities: Released MOSS-VL-Realtime — real-time video understanding on continuous streams.

🚀 Upcoming

• Training Engine: Full training code for MOSS-VL.
• RL Post-training: Reinforcement Learning for MOSS-VL series.
• Documentation: Comprehensive Technical Report.
• Cookbooks: Task-level runnable notebooks.

---

🤝 Acknowledgement

We would like to express our gratitude to NVIDIA for the Megatron-LM framework and the Qwen Team for their powerful Qwen series language models, which serve as the foundation of our training infrastructure and core LLM. We also thank the SGLang Team for their high-performance SGLang serving framework, which powers efficient deployment of MOSS-VL.

📜 Citation

@misc{moss_vl_2026,
  title         = {{MOSS-VL Technical Report}},
  author        = {OpenMOSS Team},
  year          = {2026},
  howpublished  = {\url{https://github.com/OpenMOSS/MOSS-VL}},
  note          = {GitHub repository}
}

@misc{mossvideopreview2026,
  title         = {{MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention}},
  author        = {Pengyu Wang and Chenkun Tan and Shaojun Zhou and Wei Huang and Qirui Zhou and Zhan Huang and Zhen Ye and Jijun Cheng and Xiaomeng Qian and Yanxin Chen and Xingyang He and Huazheng Zeng and Chenghao Wang and Pengfei Wang and Hongkai Wang and Shanqing Gao and Yixian Tian and Chenghao Liu and Xinghao Wang and Botian Jiang and Xipeng Qiu},
  year          = {2026},
  eprint        = {2606.07639},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2606.07639}
}

🌟 Star History

Built with ❤️ by the OpenMOSS Team