Glonce

Ovis-Embedding unifies text, image, video and audio in one embedding model

researchThe Glonce editorsHugging Face🔥74Glonce rating74Impact70Novelty68Relevance100Surprise50Players57Credibility82
Ovis-Embedding unifies text, image, video and audio in one embedding model

A research report introduces Ovis-Embedding, a family of omni-modal embedding models designed for universal, any-to-any retrieval across text, image, video and audio. Rather than stitching together separate encoder towers for each modality, the model uses a single shared multimodal backbone to map all four modalities into one common representation space, an approach the authors say avoids modality fragmentation.

The authors describe three key advances behind the system. First, native omni-modal initialization: Ovis-Embedding starts from a pretrained Qwen-omni model as its embedding backbone and adapts it through contrastive training combined with low-rank initialization. Second, data-centric omni-modal training: the team builds a broad, high-quality training corpus spanning text, images, video, audio and interleaved multimodal data, and introduces a technique called homogeneous-source sampling to assemble task-consistent batches with informative in-batch negatives, which is meant to improve data efficiency. Third, embedding-specific training and inference optimization: the model is trained with focal loss to emphasize hard examples, plus a similarity-based Embedding Distillation method that transfers fine-grained similarity structure from complementary expert models. At inference time, a low-rank feature decomposition step lets the model produce compact embeddings with flexible dimensionality while keeping performance loss minimal.

According to the report, the Ovis-Embedding family achieves state-of-the-art performance on five embedding benchmarks: MMEB-v3, MMEB-v2, MVEB, MAEB and RTEB, which together cover text, image, video and audio retrieval tasks. The report does not give specific numeric scores for any of these benchmarks, and does not name the authors, an institution, a model size, or whether code or weights are publicly released. The authors frame the results as evidence that unified omni-modal training can overcome the fragmentation that comes from training separate models per modality, and can move the field toward universal embedding models usable for any-to-any retrieval.

Key facts

  • Ovis-Embedding is an omni-modal embedding family that encodes text, image, video and audio into one shared representation space using a single backbone instead of separate modality towers.
  • It adapts a pretrained Qwen-omni model as its backbone through contrastive training with low-rank initialization.
  • Training uses homogeneous-source sampling for task-consistent batches, focal loss on hard examples, and similarity-based Embedding Distillation from complementary expert models.
  • At inference, low-rank feature decomposition produces compact embeddings with flexible dimensionality and minimal performance loss.
  • The authors report state-of-the-art results on five benchmarks: MMEB-v3, MMEB-v2, MVEB, MAEB and RTEB, without disclosing specific scores.

Why it matters

Most embedding models are built modality by modality, with separate encoders bolted together for text, image, video or audio search. Ovis-Embedding's pitch is a single shared backbone that natively handles all four at once, which the authors argue reduces the fragmentation that comes from stitching modality-specific towers together and moves toward one model usable for any-to-any retrieval.

Who it affects

The work is aimed at researchers and engineers building multimodal search and retrieval systems, such as cross-modal search (text-to-video, image-to-audio, and similar), recommendation and RAG-style retrieval pipelines that currently rely on separate per-modality embedding models.

How to use it

The report does not state model sizes, licensing terms, or whether code and weights are publicly released, so there is nothing yet to point to for direct adoption beyond the described techniques: starting from a pretrained Qwen-omni backbone, contrastive training with low-rank initialization, homogeneous-source batch sampling, focal loss, embedding distillation, and low-rank feature decomposition for compact output embeddings.

How solid is it

The claims come from the report's own empirical evaluation across five established embedding benchmarks (MMEB-v3, MMEB-v2, MVEB, MAEB, RTEB), which lends it some external grounding, but the excerpt available here gives no concrete numeric scores, no named authors or institution, and no independent verification of the state-of-the-art claim.

Risks and caveats

Without specific benchmark numbers, model size, or release details, the state-of-the-art claim cannot be checked against baselines from the source alone. The report also does not name who conducted the work or under what institution, which limits how the claims can currently be traced or reproduced.