Glonce

UltraBERT reaches 91.0 on GLUE and 47.7 on BEIR with a new encoder training recipe

researchThe Glonce editorsHugging Face🔥56Glonce rating56Impact40Novelty55Relevance95Surprise30Players15Credibility75
UltraBERT reaches 91.0 on GLUE and 47.7 on BEIR with a new encoder training recipe

A paper listed on Hugging Face Papers (2610.15028) introduces UltraBERT, a bidirectional encoder model. The authors start from a complaint about the field: recent encoders have borrowed innovations from causal large language models, but the pretraining recipe itself is still inherited from small-scale conventions. In their telling, that leaves encoders underoptimized, and no single model performs competitively on both classification and retrieval tasks.

To see how far encoder performance can be pushed, the authors revisit core training design choices and report three findings. First, encoders saturate on downstream tasks beyond a few trillion tokens, so additional compute past that point does not help. Second, across model sizes, lower masking ratios benefit classification while higher ones benefit retrieval; this contradicts prior work that suggested larger masking ratios for larger models. Third, distillation with pruning improves classification at low masking ratios but degrades retrieval at high ones.

UltraBERT is built from those findings. It assigns separate LM heads to low and high masking ratios within a single batch, and it restricts the distillation objective to the low ratio. The authors write that, to their knowledge, UltraBERT is the first encoder to achieve state of the art on both classification and retrieval tasks, reaching 91.0 on GLUE and 47.7 on BEIR. They add that the results show how far encoder performance can be pushed and give a principled basis for compute allocation, masking ratio design and knowledge transfer in bidirectional models.

Key facts

  • UltraBERT is a single bidirectional encoder that, per the authors, reaches 91.0 on GLUE (classification) and 47.7 on BEIR (retrieval).
  • The authors say that, to their knowledge, it is the first encoder to reach state of the art on both classification and retrieval.
  • Design: separate LM heads for low and high masking ratios within a batch, with the distillation objective restricted to the low ratio.
  • Finding 1: encoders saturate on downstream tasks beyond a few trillion training tokens, so extra compute past that point does not help.
  • Finding 2: lower masking ratios help classification and higher ones help retrieval across model sizes, against prior work suggesting larger ratios for larger models.

Why it matters

Encoders have been the workhorse for classification and retrieval, but the authors argue their pretraining recipes were never properly tuned and that no single model has been competitive on both kinds of task. UltraBERT is presented as one model that closes that gap. The paper also makes concrete claims about training practice: stop scaling tokens past a few trillion, and treat the masking ratio as a task-dependent choice rather than something that simply grows with model size.

Who it affects

People who pretrain bidirectional encoders, and teams that run separate encoders for classification and for retrieval today. The findings on masking ratio and distillation are aimed at anyone deciding how to allocate training compute for encoder models.

How to use it

The source gives no information on usable artifacts, so there is nothing concrete to download or run from this text; the source does not say whether the model weights or code are released. What it does offer is a set of design lessons: use lower masking ratios when classification matters and higher ones when retrieval matters, train both in the same batch with separate LM heads, and apply the distillation objective only at the low ratio.

How solid is it

These are the authors' own results, reported in a paper abstract: 91.0 on GLUE and 47.7 on BEIR. The claim of being first to reach state of the art on both task families is hedged by the authors themselves with "to our knowledge". No authors or institutions are named in the text, and no baseline models or their scores are given, so no improvement margin can be stated from this source.

Risks and caveats

The source does not say which GLUE or BEIR variant, subset or aggregation the two scores refer to. It gives no model size, parameter count or exact training-token count for UltraBERT, and no values for what counts as a "low" or "high" masking ratio. The saturation finding is stated only as "a few trillion tokens". Without those details, the headline numbers and the findings are hard to reproduce or compare from this text alone.

“To our knowledge, UltraBERT is the first encoder to achieve state of the art on both classification and retrieval tasks, reaching 91.0 on GLUE and 47.7 on BEIR.”

— UltraBERT paper abstract