VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding

As video understanding becomes central to real-world AI applications, current models face a trilemma: they struggle to generalize across diverse video types, they are computationally expensive to scale, and they are often closed-source or lack reproducible training data. This paper asks: can we build a fully open, efficient, and generalist video MLLM that balances high-quality perception with low computational overhead?

The authors introduce VideoChat3, a 4B-parameter model built on three pillars. First, they propose the Inflated 3D Vision Transformer (I3D-ViT), which replaces standard frame-by-frame encoding with a chunk-wise spatiotemporal attention mechanism. This allows the model to process video chunks as unified units, effectively modeling motion while reducing redundant visual tokens. Second, they implement an Adaptive Frame Resolution strategy for streaming video, where the model dynamically adjusts its pixel quota based on the perceived importance of the video content, allowing it to monitor streams at low cost and focus high-resolution attention only when necessary. Finally, they curate three large-scale, high-quality datasets (Academic2M, LV116K, and OL617K) and release the entire training pipeline, including weights and data, to ensure full reproducibility.

VideoChat3 demonstrates that efficient architectural design can outperform larger, closed-source models. By compressing video representations at the tokenizer level, the model achieves a 16x spatiotemporal compression ratio, significantly reducing the computational burden of long-form and streaming video tasks. Experimental results show that VideoChat3 surpasses prior open-source models of comparable or larger sizes across a wide range of benchmarks, including temporal grounding, long-video QA, and online streaming perception. The model’s ability to proactively manage its own inference state—deciding when to observe, wait, or respond—marks a significant step toward real-world, interactive video agents.