Glonce

Youtu-Parsing-Omni, a 5B parser for documents, audio and video, tops OmniDocBench v1.6

researchThe Glonce editorsHugging Face🔥62Glonce rating62Impact50Novelty60Relevance100Surprise35Players25Credibility65
Youtu-Parsing-Omni, a 5B parser for documents, audio and video, tops OmniDocBench v1.6

A new paper presents Youtu-Parsing-Omni, an omni-modal parser. It converts documents, images, audio and video into structured records grounded in space and time. The authors pitch it as a perception layer for agents that need to read, locate and cite evidence across modalities.

The design rests on two ideas: one encoder for every modality and one schema for every task. The encoder, Youtu-Omni-Encoder, is initialized from a pre-trained language model to strengthen text reading. It encodes all modalities with shared weights in a unified space-time coordinate system. A few fusion layers let video frames attend to concurrent audio, which the authors say brings cross-modal fusion and temporal alignment into perception itself.

On the output side, a schema called OmniSchema unifies the outputs of all ten parsing tasks. That makes it possible to train with a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), which gives all tasks a shared token-role-based objective. In each round the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself. That copy is then refreshed with the trained weights, so the parser and its supervision co-evolve.

With a single 5B-parameter checkpoint, the model reports state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall). It leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also reports the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems. In character-level transcription of typeset sheet music it matches a dedicated optical-music model while staying far ahead of general-purpose systems. Code and model weights are publicly available.

Key facts

  • Youtu-Parsing-Omni converts documents, images, audio and video into structured records grounded in space and time, using one shared encoder and one output schema (OmniSchema) covering ten parsing tasks.
  • A single 5B-parameter checkpoint reports state-of-the-art on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with the advantage concentrated on audio and video.
  • Training uses OmniSchema-Aware On-Policy Distillation (OSAD): the model learns from its own rollouts against token-level targets from a reference-conditioned copy of itself, which is refreshed each round.
  • Other reported results: best overall chart-parsing score (95.02), highest exact chemical-fingerprint agreement among compared systems, and parity with a dedicated optical-music model on typeset sheet music.
  • Code and model weights are publicly available.

Why it matters

Most parsers handle one kind of input: a document reader, a speech transcriber, a video tool. This paper argues for one model and one output format across all of them. The authors present it as a perception layer for agents that must read, locate and cite evidence, so that an agent gets grounded records in space and time rather than loose text. The approach also folds cross-modal fusion, such as video frames attending to concurrent audio, into the perception step itself.

Who it affects

The most direct audience is people building agents or pipelines that need structured, citable output from mixed material: scanned documents, charts, recordings and video. Those working on specialised parsing, such as chemical structures or sheet music, are also covered by the reported results. Because the weights are public, researchers can test the claims themselves.

How to use it

The paper states that code and model weights are publicly available. The checkpoint is a single 5B-parameter model, and all ten parsing tasks share one output schema, OmniSchema, so one model can serve several tasks that would otherwise need separate tools.

How solid is it

The numbers are the authors' own: 96.96 Overall on OmniDocBench v1.6, 75.08 Avg. on OmniParsingBench, 95.02 for chart parsing. The abstract gives no baseline names or scores for competing systems, so the margins over them are unknown. The numbers for chart parsing, chemical fingerprint and sheet music are not tied to named benchmarks, and no score is given for the chemical-fingerprint or sheet-music results. The OmniParsingBench claim is limited to open-weight models; it does not say the model beats closed-weight systems there.

Risks and caveats

These are self-reported results from the paper's abstract. The abstract gives no training data, compute or inference speed figures, so cost and practical speed are unknown. The sheet-music result is a match with a dedicated model, not a win. The chemical-fingerprint claim holds only among the systems the authors compared. Independent evaluation would be needed to confirm the state-of-the-art claims.