Glonce

NeoMME encoder matches ColQwen2.5 in document retrieval with 14× fewer parameters

modelsThe Glonce editorsHugging Face🔥78Glonce rating78Impact70Novelty75Relevance100Surprise60Players65Credibility90
NeoMME encoder matches ColQwen2.5 in document retrieval with 14× fewer parameters

NeoMME is a new family of efficient multimodal encoders released in two sizes, 260M and 800M parameters, described in a Hugging Face blog post. Unlike many recent visual document retrieval models, which combine a separately pretrained vision tower with a causal language model, NeoMME does not use either: a single bidirectional Transformer processes both text tokens and raw image patches, and the whole model is trained from scratch with a masked discrete-diffusion objective rather than adapted from an existing text or vision model. All model checkpoints are released under the Apache 2.0 license and are available through Hugging Face Transformers.

Many recent visual document retrievers adapt pretrained generative vision-language models: a separately pretrained vision encoder feeds visual features through a projector into a causal language model, which then processes the combined representation. But retrieval, classification and token labelling do not generate text autoregressively, so, the authors argue, none of them actually need a causal decoder or the extra parameters and compute that architecture carries. ModernBERT had already brought efficiency gains to bidirectional text encoders, and ModernVBERT extended that approach to visual retrieval while still relying on a separate, pretrained SigLIP2 vision tower. NeoMME goes further, training a single multimodal encoder from scratch without carrying over a VLM's parameter and compute overhead at all.

Both sizes share the same architecture. Text enters through factorized token embeddings, while images are split into a grid of non-overlapping 32×32 patches and projected with a small MLP; both modalities then feed into the same Transformer. Images keep their original aspect ratio and size, so the model spends more tokens on a dense, high-resolution document page than on a smaller, sparser image. Context length is 16,384 tokens for both sizes, enough to hold up to two standard 3840×2160 4K UHD images; most layers use symmetric sliding-window attention, while every sixth layer and the final layer use global attention. The stack also carries a number of recent encoder-architecture improvements: grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings and squared-ReLU MLPs. The tokenizer, too, is built from scratch: a BPE (byte-pair encoding) vocabulary of 131,000 tokens trained on multilingual text, code, mathematics and machine-produced image transcripts.

NeoMME is pretrained from scratch as a discrete masked-diffusion text denoiser. For text-only examples, a corruption rate is sampled uniformly between 0 and 1, and each eligible token is masked independently at that rate; for multimodal examples the corruption rate is instead sampled between 0.3 and 1, while the image patches themselves stay fully visible. With light masking the model can often recover a missing word purely from surrounding text, the way 'cat' is a plausible fill for 'The [MASK] sat on the mat' even with no image present; heavy masking instead forces it to lean on the image, since the surviving text carries little to no signal. Pretraining data mixes multilingual text, code, mathematics, natural images and document images. Each model size sees about 524 billion packed input tokens in total, of which 290 billion come from text-only examples, a notably smaller budget than ModernBERT's 2 trillion training tokens; the authors credit the NorMuon optimizer with recovering some of that data efficiency.

The authors fine-tuned NeoMME into NeoMME-Retriever for visual document retrieval, following the page-image approach introduced by ColPali: instead of extracting and chunking text from a PDF, the retriever ranks screenshots of whole document pages, preserving layout, tables, charts, fonts and other visual cues that even a perfect OCR pipeline would lose. NeoMME-Retriever adds two jointly trained heads on top of the shared backbone. A dense head mean-pools the backbone's hidden states into one normalized vector, compact and well suited to fast approximate-nearest-neighbour search. A late-interaction head instead projects every individual text token or image patch into its own 128-dimensional normalized vector, preserving finer local matches between a query's tokens and specific regions of a page. Omar Khattab, who introduced late-interaction in the ColBERT retrieval model, is credited in the post with explaining why the term is more precise than 'multi-vector': it describes the granularity and learnability of the scoring function, not just how many vectors get stored. The post separately points readers to a crash course on late-interaction written by Amélie Chatelain. A single NeoMME-Retriever forward pass returns both dense and late-interaction embeddings at once. The authors recommend using late-interaction generally, since it is more powerful and works with open-source libraries such as NextPlaid, but suggest a two-stage pattern for very large corpora: retrieve an initial candidate set cheaply through the dense embedding and an ANN index, then rerank that shortlist with late-interaction.

On the ViDoRe v3 benchmark, measured as nDCG@10, NeoMME-Retriever-260M scores 0.523, the highest result among all evaluated models under 800M parameters, and comes within 0.002 of ColQwen2.5 despite using about 14 times fewer parameters. The 800M version scores 0.556, within 0.009 of the similarly sized Vultron Retriever Flash (0.8B); both NeoMME-Retriever sizes sit on the Pareto frontier for score versus model size. On the older ViDoRe v1 and v2 benchmarks, measured as nDCG@5, NeoMME-Retriever-260M beats ColModernVBERT and the twice-larger ColSmol-500M, while NeoMME-Retriever-800M beats ColPali v1.3 while using 3.6 times fewer parameters.

Late-interaction storage scales with the number of vectors per page, so higher-resolution scans cost more to index: a single 2048×2048 page produces 4,200 vectors from NeoMME-Retriever, about 2.1 MB in float32, and the measured average across ViDoRe v3 is about 1.5 MB per document. To cut that down, the authors combine two compression methods: hierarchical token pooling, which clusters similar vectors within a page's embedding and replaces each cluster with its mean, and asymmetric quantization, which compresses stored document embeddings to int8 or binary precision while keeping query embeddings, generated on the fly rather than stored, at higher precision. With a pooling factor of 10 and int8 precision for both queries and documents, per-page storage falls from about 1.5 MB to 39 kB, a 39 times reduction, while keeping more than 99% of the uncompressed nDCG@10. A more aggressive setting, pooling factor 8 with int8 queries and binary documents, cuts storage to 6 kB per page, 255 times smaller than the baseline, while still retaining more than 95% of the original retrieval quality.

The authors also measured image-encoding throughput against other multimodal document retrievers, using preprocessed image tensors with a batch size calibrated separately for each model and image size. At a matched 2048×2048 input on a single NVIDIA L40S GPU, NeoMME-Retriever-260M encodes about 51 pages per second, nearly twice ColModernVBERT's 26 pages per second; both the 260M and 800M NeoMME-Retriever models are also described as faster than the other retrievers tested at smaller input sizes, though no specific figures are given for those cases.

Both sizes are available through Hugging Face Transformers, with example code showing how to load the model and its processor and score text queries against document-page images using either dense cosine similarity or late-interaction (MeanMaxSim) scoring.

Key facts

  • NeoMME ships in 260M and 800M sizes: one bidirectional Transformer processes both text tokens and raw image patches, replacing the separate vision tower and causal decoder most visual document retrievers use, and the whole model trains from scratch with a masked discrete-diffusion objective.
  • Fine-tuned into NeoMME-Retriever for page-image document retrieval, the 260M model scores 0.523 nDCG@10 on ViDoRe v3, within 0.002 of ColQwen2.5 while using about 14 times fewer parameters.
  • The 800M model scores 0.556 nDCG@10 on ViDoRe v3, within 0.009 of the similarly sized Vultron Retriever Flash; both sizes land on the Pareto frontier of score versus parameter count.
  • Hierarchical token pooling combined with int8 or binary quantization cuts the late-interaction storage index from about 1.5 MB to as little as 6 kB per page, 255 times smaller, while keeping over 95% of baseline retrieval quality.
  • At matched 2048×2048 input on one NVIDIA L40S GPU, the 260M retriever encodes about 51 pages per second, nearly twice ColModernVBERT's 26; every checkpoint is released under the Apache 2.0 license.

Why it matters

Visual document retrieval, ranking whole page screenshots so a search system can skip OCR and text extraction entirely, has mostly been built on pretrained generative vision-language models, which carry a full causal decoder even though retrieval never generates text. NeoMME instead trains a single bidirectional encoder from scratch for both modalities, and its retriever variant reaches results within a fraction of a percentage point of models built on far bigger backbones: ColQwen2.5, which NeoMME-Retriever-260M nearly matches, is about 14 times larger. Late-interaction retrieval's other practical obstacle has been index size, since it stores a separate vector for every token or image patch on every page rather than one vector per document. The compression setup described here takes per-page storage from about 1.5 MB down to 6 kB, a 255-fold reduction, while keeping over 95% of retrieval quality, addressing that obstacle directly, at model sizes that keep both training and inference cheaper than the vision-language-model-based alternatives it competes with.

Who it affects

Anyone building search over scanned or visually dense documents, forms, tables, charts, multi-column PDFs, where converting pages to plain text through OCR throws away structure, is the direct audience: NeoMME-Retriever ranks page images directly instead of extracted text. The parameter savings matter to teams running retrieval at scale, since fewer parameters and faster encoding, about 51 pages per second for the 260M retriever versus ColModernVBERT's 26, both on one NVIDIA L40S GPU, cut the GPU time needed to index a large document set. The storage compression is aimed squarely at anyone running late-interaction retrieval in production: it makes a per-token, per-patch vector index cheap enough to keep in an ordinary vector database such as Qdrant, Weaviate or Milvus. Multilingual training data and an Apache 2.0 license on every checkpoint also open the model to non-English deployments and commercial use without licensing negotiations.

How to use it

Both NeoMME and NeoMME-Retriever, in the 260M and 800M sizes, are available through Hugging Face Transformers, installed from the project's git main branch alongside sentence-transformers 6.0 or later; the 260M retrieval checkpoint is published under the model id Hcompany/NeoMME-260M-Retriever. Loading NeoMMEProcessor and NeoMMEForRetrieval for that id, encoding document-page images with the 'document' task and text with the 'query' task, and running a single forward pass returns both a dense embedding and a late-interaction embedding together. Dense matches are scored with cos_sim from sentence-transformers; late-interaction matches use mean_maxsim. The post recommends late-interaction scoring by default, pairing it with the open-source NextPlaid library, and reserves the cheaper dense-embedding-plus-ANN route for pulling an initial candidate set out of a very large corpus before reranking that shortlist with late-interaction. Every checkpoint ships under the Apache 2.0 license, free to adapt and deploy commercially.

How solid is it

The retrieval numbers come from three established visual-document-retrieval benchmarks, ViDoRe v1, v2 and v3, rather than a benchmark the authors built themselves, and the post marks which comparison figures are drawn from the MTEB leaderboard versus which come from its own evaluations, a level of sourcing transparency retrieval write-ups do not always provide. Several things a full assessment would want are missing, though: no training hardware, cluster size or wall-clock training time is disclosed anywhere in the post, no publication date or named author or team appears in the prose, and while classification and token labelling are cited as motivating use cases for the non-causal design, the only benchmark numbers reported are for retrieval; neither of those other tasks is measured here. The named comparison models, ColQwen2.5, ColModernVBERT, ColSmol-500M, ColPali v1.3 and Vultron Retriever Flash, are all established entries in the same visual-retrieval space, which makes the head-to-head scores meaningful, though a reader cannot check them directly without the underlying ViDoRe leaderboard.

Risks and caveats

Compression is not free: the most aggressive setting, 6 kB per page via pooling factor 8 with int8 queries and binary documents, gives up close to 5% of retrieval quality for its 255-fold storage cut, and the milder 39 kB setting trades less compression for a smaller quality hit, so a deployment has to pick a point on that frontier rather than getting maximum compression at no cost. The 16,384-token context caps a single forward pass at roughly two standard 4K page images, so very large scans or long multi-page documents may need to be split before encoding. The claim that both model sizes are faster than the compared models at smaller image resolutions is stated without the supporting numbers given for the matched 2048×2048 case, so it cannot be checked against figures in the post itself. And the version of the post captured here cuts off partway through the usage-example code, so any closing remarks, caveats or citation details on the live page are not reflected in this summary.