Trainable [DUMP] prefix tokens cut noise in retrieval embeddings, paper says

Embedding models turn text of any length into a fixed-size vector. The authors of a new paper argue that this compression can entangle the meaning of the text with leftover token-level information, such as artifacts that come from attention sink behavior. That leftover signal ends up in the final retrieval representation, where it does not belong.

Their fix is simple. During embedding model training, they prepend trainable [DUMP] tokens to the input. The model architecture, the pooling strategy and the training loss all stay unchanged. The added tokens are optimized through the standard contrastive retrieval objective, like the rest of the training.

The idea is that these tokens act as trainable prefix positions that interact with the regular tokens while the text is encoded. According to the authors, this helps reduce the influence of residual or uninformative token-level signals on the final retrieval representation.

The authors tested the method on standard retrieval and long-context benchmarks. They report that it consistently improves over finetuned baselines, and that this holds across both encoder and decoder embedding models. A further analysis, they say, shows that the [DUMP] tokens alter token-level activation and attention patterns associated with low-information tokens.

From this, the authors conclude that trainable prefix tokens offer a lightweight way to improve retrieval representations. The text available here gives no numeric results, benchmark names or model names, so the size of the gains is not known.

Key facts

The method prepends trainable [DUMP] tokens to the input while training embedding models.
Model architecture, pooling strategy and training loss are unchanged; the tokens are optimized through the standard contrastive retrieval objective.
The motivation: compressing text into a fixed-size vector can entangle semantic content with residual token-level information, such as attention sink artifacts.
The authors report consistent gains over finetuned baselines on retrieval and long-context benchmarks, for both encoder and decoder embedding models.
Analysis reportedly shows the tokens change activation and attention patterns tied to low-information tokens.

Why it matters

Retrieval systems depend on embeddings, and the quality of one fixed-size vector per text decides what gets found. This paper points at a specific weakness: leftover token-level signals, including attention sink artifacts, can mix into the representation. Its answer needs no redesign of the model, only extra trainable tokens at the start of the input.

Who it affects

People who train or finetune embedding models for retrieval are the direct audience. The authors say the method works with both encoder and decoder embedding models, so it is not tied to one model family.

How to use it

As described, the recipe has one step: prepend trainable [DUMP] tokens to the input during embedding model training and optimize them with the usual contrastive retrieval objective. Architecture, pooling and loss do not change. The text gives no token count and mentions no code or model release, so a reader would need the full paper for implementation details.

How solid is it

This is a preprint listed on Hugging Face Papers, and the claims are the authors' own. They report consistent improvements over finetuned baselines on standard retrieval and long-context benchmarks. The text names no benchmarks or models and gives no scores, so the strength of the evidence cannot be judged from it. The only comparison mentioned is against finetuned baselines.

Risks and caveats

The authors say the results suggest the approach is lightweight, but the text gives no training cost or inference overhead figures. No comparison with other methods beyond finetuned baselines is mentioned. The size of the gains is not stated, and independent replication is not described.