Youtu-Parsing-Omni Team
8 min
Abstract
We present Youtu-Parsing-Omni, an omni-modal parser that converts documents, images, audio, and video into structured records grounded in space and time, providing a perception layer for agents that read, locate, and cite evidence across modalities. Its design rests on one encoder for every modality and one schema for every task. Youtu-Omni-Encoder, initialized from a pre-trained language model to strengthen text reading, encodes all modalities with shared weights in a unified space-time coordinate system, and a few fusion layers let video frames attend to concurrent audio, bringing cross-modal fusion and temporal alignment into perception itself. OmniSchema unifies the outputs of all ten parsing tasks, allowing a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), to provide a shared token-role-based objective across tasks. In each round, the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself, which is then refreshed with the trained weights, so that the parser and its supervision co-evolve. With a single 5 B-parameter checkpoint, Youtu-Parsing-Omni achieves state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also achieves the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems, and matches a dedicated optical-music model in character-level transcription of typeset sheet music while staying far ahead of general-purpose systems. Code and model weights are publicly available.
Alex: The report gives the example of identifying which words are spoken while a particular slide is visible. A correct video record needs both the words and their correspondence to the slide. This model combines concurrent sound and frames within short windows; relationships across windows remain the language model's job.
Sam: And after that shared processing, what does the researcher actually receive?
Alex: A text record following OmniSchema, their common output format, covering ten parsing tasks. An image element gets a bounding box and extracted content. An audio or video segment gets start and end times, with speech linked to a speaker identifier.
Sam: A common format could still flatten everything into a generic caption. Do chemical structures and music retain their formal content?
Alex: They do. Molecules are written in SMILES, a text notation for chemical structures. Music becomes ABC notation, a text representation of a score. The shared format holds those task-specific contents alongside their source anchors; it doesn't replace them with prose.
Sam: Producing that structure reliably sounds like a different problem from recognizing the input. How do they train for both?
Alex: They first build perception through staged training, gradually allowing larger inputs and audio-visual interaction. Supervised fine-tuning then teaches the required output format. After that, reinforcement learning scores the model's generated records for validity and task-specific accuracy.
Sam: A whole-record score seems blunt if the mistake is one formula symbol or one table entry. Is that where their other training contribution comes in?
Alex: Yes: on-policy distillation, meaning teacher feedback on outputs the learner itself generates. A frozen copy of the model acts as teacher, but it also sees the reference answer. It evaluates the student's next-word choices along the student's own generated record.
Sam: The teacher has extra information, rather than being a separately trained, stronger model. How does the common format help?
Alex: It lets training distinguish content from structure. A description may allow several valid wordings, while punctuation and field names have tighter constraints. The method treats those roles differently, then periodically replaces the teacher with the improved student.
Sam: Let's turn to the evidence. Does broad coverage come at the expense of their original strength in documents?
Alex: On OmniDocBench, the reported overall score is 96.96, compared with TeleOCR's 96.91. That's a narrow lead, not a practical gulf. The meaningful result is that the same model remains competitive with document specialists while handling other modalities.
Sam: That establishes breadth without an obvious document penalty. Does it also outperform the other broad parsers?
Alex: On their OmniParsingBench evaluation, its average is about seventy-five, versus about seventy-two for Logics-Parsing-Omni. It leads the open-weight models evaluated across the full suite. Gemini's proprietary model still scores higher overall.
Sam: An average could hide a lot. Where does the advantage actually show up?
Alex: Mainly in audio and video, especially locating events in time. It also has the strongest chart score among the compared systems. But it trails Logics on natural images and geometry, so this isn't uniform dominance.
Sam: And better audio parsing doesn't necessarily mean better speech recognition. Do those move together here?
Alex: They don't. On the audio split, its speech-transcription score is the weakest among the compared models. Better timing and event descriptions compensate. Likewise, the chemical and music results are metric-specific: strongest exact chemical-fingerprint agreement, and specialist-level character transcription of typeset music, not superiority on every measure.
Sam: Before treating those rankings as definitive, how comparable were the evaluations?
Alex: The broad parsing comparison includes about thirty-seven hundred evaluated samples. The authors built their own scoring toolkit and ran every compared system through it. They repair formatting errors before scoring, exclude a camera-video subset with annotation problems, and remove unavailable media.
Sam: So the internal comparison is consistent, but someone shouldn't line those scores up with the benchmark's original leaderboard.
Alex: The report explicitly says they aren't directly comparable. There's another boundary: the gains fit the shared-timeline argument, but comparisons between different complete systems don't isolate that architecture's contribution. The controlled encoder experiments instead show a reading benefit from starting with language-model weights, not a general spatial-reasoning benefit.
Sam: Do they have clearer evidence for the final distillation stage?
Alex: It accounts for eighty-eight percent of the document-score improvement beyond supervised fine-tuning. That's substantial evidence for the complete stage's contribution. It doesn't, by itself, separate the effects of its role-specific feedback and teacher refresh.
Sam: What would make you hesitate before putting this into a working extraction pipeline?
Alex: Throughput. Unlike its predecessor, this model doesn't yet use parallel decoding acceleration. It generates the record sequentially with a larger decoder, so the authors expect lower throughput on dense material such as tables.
Sam: Given those strengths and trade-offs, who should read the full report, and where should they begin?
Alex: Researchers building evidence-reading agents or multimodal extraction systems should read it. Start with Omni-Modal Parsing Schema to see whether the output fits your needs. Then read the OmniParsingBench evaluation protocol and the ablation study before interpreting the rankings.
Sam: And for researchers who don't need another parser in their pipeline, what's the point to carry away?
Alex: Useful extraction isn't just what the source says. It's what it says, linked to where and when it happened.
Sam: Keep the source attached to the answer.