Glonce

NeoMME ships a single-tower encoder with about 2x retrieval speed

modelsThe Glonce editorsHugging Face🔥67Glonce rating67Impact65Novelty60Relevance95Surprise35Players40Credibility85
NeoMME ships a single-tower encoder with about 2x retrieval speed

Visual document retrievers such as ColPali are usually built by repurposing architectures designed for generative vision-language modeling: a separately pretrained vision encoder bolted onto a causal language model. That approach carries over the full parameter and compute overhead of a generative VLM even though retrieval itself is not a generative task.

NeoMME takes a different route. It is a family of two bidirectional Transformer encoders, at 260M and 800M parameters, that process multilingual text and raw image patches inside a single tower rather than combining separate components. Both sizes are pretrained from scratch with a masked discrete-diffusion text objective, conditioned on visible image patches for multimodal examples, and both support a 16,384-token context, enough to encode up to two standard 4K UHD images at once.

To test downstream performance, the authors fine-tuned NeoMME with jointly trained dense and late-interaction retrieval heads. On the ViDoRe v3 benchmark, NeoMME-Retriever 260M reaches 0.523 nDCG@10, ahead of every evaluated model strictly below 800M parameters, while NeoMME-Retriever 800M reaches 0.556. At a matched 2048x2048 image input size on an NVIDIA L40S GPU, NeoMME-260M encodes pages at about 2x the throughput of ColModernVBERT.

A separate efficiency step targets storage rather than speed: hierarchical token pooling combined with asymmetric quantization compresses late-interaction multimodal document embeddings by 255x while preserving over 95% of the baseline nDCG@10 score.

The authors contributed NeoMME to Hugging Face Transformers and released the pretrained backbone and retrieval-compatible checkpoints under the Apache 2.0 license, published in the Hcompany collection on Hugging Face.

Key facts

  • NeoMME comes as 260M and 800M-parameter single-tower bidirectional Transformer encoders that process multilingual text and raw image patches together, instead of pairing a separate vision encoder with a language model.
  • Both sizes are pretrained from scratch with a masked discrete-diffusion text objective and support a 16,384-token context, enough for up to two standard 4K UHD images.
  • Fine-tuned with dense and late-interaction retrieval heads, NeoMME-Retriever 260M scores 0.523 nDCG@10 on ViDoRe v3 (best under 800M parameters), and NeoMME-Retriever 800M reaches 0.556.
  • At a matched 2048x2048 input size on an NVIDIA L40S, NeoMME-260M encodes pages at about 2x the throughput of ColModernVBERT.
  • Hierarchical token pooling and asymmetric quantization compress late-interaction embeddings by 255x while keeping over 95% of baseline nDCG@10; the backbone and retrieval checkpoints are released under Apache 2.0.

Why it matters

Visual document retrieval systems such as ColPali repurpose architectures built for generative vision-language modeling, combining a separately pretrained vision encoder with a causal language model, as retrieval encoders. That inherits the full parameter and compute overhead of a generative VLM for a task that is not generative at all. NeoMME instead builds a single bidirectional Transformer that natively takes multilingual text and raw image patches, removing the redundant overhead at the architecture level rather than trimming it after the fact.

Who it affects

Teams building visual document search and retrieval-augmented pipelines over multilingual document collections, since NeoMME reads text and image patches inside one native multimodal encoder rather than a bolted-together generative VLM. Two sizes, 260M and 800M parameters, let a team pick between smaller footprint and higher accuracy.

How to use it

The pretrained backbone and retrieval-compatible checkpoints are released under the Apache 2.0 license, contributed to Hugging Face Transformers and published in the Hcompany collection on Hugging Face. Both sizes support a 16,384-token context, enough to encode up to two standard 4K UHD images in a single pass, and can be fine-tuned with dense and late-interaction retrieval heads the way the authors did for the ViDoRe v3 evaluation.

How solid is it

The retrieval results come from the ViDoRe v3 benchmark: NeoMME-Retriever 260M reaches 0.523 nDCG@10, the best score among evaluated models strictly below 800M parameters, and NeoMME-Retriever 800M reaches 0.556. The throughput claim, about 2x versus ColModernVBERT, was measured at a matched 2048x2048 image input size on a single NVIDIA L40S GPU. The source names ColPali and ColModernVBERT specifically but does not list the other models NeoMME-Retriever 260M outperforms under 800M parameters, and it gives no absolute pages-per-second figures, only the relative 2x number.

Risks and caveats

The source gives no author names or institutions, no training data sources or sizes, no compute budget for pretraining, and no release timeline beyond the paper existing. The reported gains sit on one benchmark family (ViDoRe v3) and one GPU target (NVIDIA L40S); performance on other retrieval benchmarks or hardware is not stated.