Glonce

Hugging Face shows pooling-aware training keeping 97.8% of retrieval quality at 4x compression

researchThe Glonce editorsHugging Face🔥56Glonce rating56Impact42Novelty55Relevance95Surprise25Players15Credibility75
Hugging Face shows pooling-aware training keeping 97.8% of retrieval quality at 4x compression

Multi-vector embedding models such as ColBERT give every query token a chance to match something in a document, which makes them expressive for search. The cost is storage: a corpus of millions of documents can hold hundreds of millions of vectors. A Hugging Face blog post, written in the first person, walks through one way to shrink that footprint, token pooling, and shows how to train a model so it survives the compression.

The post gives an illustrative storage calculation. With 4-bit residual quantization, as supported by Fast-PLAID, 200 vectors of 128 dimensions need 12.8 KB of quantized residuals per document, or 12.8 GB across a million documents. Pooling each document down to 50 vectors, at the same dimensionality and precision, cuts that to 3.2 GB. These figures are arithmetic, not measurements of the trained model.

Token pooling merges similar token vectors. The HierarchicalTokenPooling module groups token embeddings with Ward hierarchical clustering: it starts with one group per vector and repeatedly merges the two groups that least increase the spread (sum of squared distances to the group mean) inside the groups. Each final group is represented by its mean, and the vectors are normalized again for MaxSim scoring. With pool_factor=4, a document keeps roughly a quarter of its vectors, so longer documents keep proportionally more. By default the first token is protected from merging, and pooling applies only to document embeddings, not queries. A distance_threshold option instead lets the vector count grow when a document has more distinct embeddings. This adaptive pooling works with any multi-vector model, including one not trained for pooling, and the post shows how to calibrate the threshold on a sample of the corpus to hit a target average vector count (the example uses 16, as an average, so individual documents can get more or fewer).

The problem is that merging can blur a distinction that matters for retrieval, and ordinary training never asked the model to cope with compressed representations. Pooling-aware training does. Much like Matryoshka embeddings, which apply a retrieval objective at several embedding widths, it applies the objective to representations with fewer token vectors. In the example, each batch contributes two losses, averaged: retrieval with the original document embeddings, and retrieval after compressing documents to about a quarter of their vectors. Because Ward clustering is discrete, training uses a straight-through estimator to pass gradients from the pooled vectors back to their member tokens, without differentiating the cluster assignments.

In Sentence Transformers this is done with MultiVectorTokenPoolingLoss, which wraps an existing multi-vector retrieval loss. The example starts from answerdotai/ModernBERT-base with MaxSim similarity, wraps CachedMultiVectorMultipleNegativesRankingLoss, and passes token_poolings=[None, pooling] with weights=[0.5, 0.5]. Gradient caching lets the encoder process mini-batches of 64 while keeping a training batch size of 256 for in-batch negatives. Extra poolers reuse the computed token embeddings, so each adds clustering and scoring work but no separate encoder pass. Custom BaseTokenPooling implementations are possible if they preserve gradients, and several poolers with their own weights can be combined.

The author's headline result: with pool_factor=4, a normally trained model retains 94.6% of its unpooled NanoBEIR nDCG@10 with a 4x reduction in document vectors, and token pooling-aware training raises that retention to 97.8% at the same storage budget.

The post also covers practical setup. The training script can be run locally with uv, with or without a --baseline flag for the ordinary-loss variant, or on Hugging Face Jobs on an A10G GPU (flavor a10g-small) with a four-hour limit, uploading the model to the user's Hub account. Optional kernels speed up MaxSim scoring, Ward clustering and some backbone operations. The default MaxSim kernel avoids building the full token similarity matrix in memory but needs CUDA and Triton, and the default Ward kernel runs on CUDA and is 30-150x faster than SciPy depending on the setting. Kernel settings are not saved with the model, so use_kernels=True has to be passed again at load time. The post credits Stefan Josef's Learn to Pool and LightOn's pooling regularization study, which also explores targeted training and adaptive budgets.

Key facts

  • At pool_factor=4 (4x fewer document vectors), a normally trained model keeps 94.6% of its unpooled NanoBEIR nDCG@10; the pooling-aware model keeps 97.8% at the same storage budget.
  • Training averages two losses with weights 0.5 and 0.5: retrieval on the original document embeddings and retrieval after pooling to about a quarter of the vectors, using MultiVectorTokenPoolingLoss in Sentence Transformers.
  • A straight-through estimator passes gradients from pooled vectors back to member tokens, since Ward clustering assignments are discrete.
  • In the post's illustrative calculation, 200 vectors of 128 dimensions at 4 bits cost 12.8 GB per million documents; pooling to 50 vectors cuts that to 3.2 GB.
  • The post builds on Stefan Josef's Learn to Pool and LightOn's pooling regularization study, and the example runs on an A10G GPU through Hugging Face Jobs.

Why it matters

Multi-vector retrieval is accurate but storage-hungry, since every document carries hundreds of token vectors. Token pooling shrinks that, but the more aggressively vectors are merged, the easier it is to lose something useful. The post's claim is that training the encoder with pooling in the loop recovers much of the lost quality: 97.8% retention against 94.6% at the same 4x reduction. The post builds on earlier work by Stefan Josef and LightOn rather than introducing the idea from scratch, and it focuses on a short, reproducible Sentence Transformers example.

Who it affects

Teams running or building ColBERT-style multi-vector search who pay for index size, and anyone training such models with Sentence Transformers. The adaptive threshold pooling also matters to people who only use an existing multi-vector model, because the post says it works with models not trained for pooling.

How to use it

Wrap a multi-vector retrieval loss, for example CachedMultiVectorMultipleNegativesRankingLoss, in MultiVectorTokenPoolingLoss with token_poolings=[None, HierarchicalTokenPooling(pool_factor=4)] and weights=[0.5, 0.5]. Replace the factor-4 pooler with your own deployment configuration, or add several poolers with their own weights. To try the post's script, run training_cached_contrastive.py with uv locally (add --baseline for the ordinary-loss version) or launch it on Hugging Face Jobs with an A10G GPU and a four-hour timeout, passing HF_TOKEN as a secret so the model can be uploaded. For an existing model, calibrate_distance_threshold finds a threshold for a target average vector count on a sample of your documents. Kernels need the sentence-transformers[train,kernels] install and use_kernels=True, which must be set again when loading the model for inference.

How solid is it

The numbers are the author's own results from a single worked example, and they are retention figures relative to the unpooled model's NanoBEIR nDCG@10; the absolute nDCG@10 values are not given in the visible text. The visible text does not say which training dataset was used or whether the 94.6% and 97.8% figures come from one run or several. The 12.8 KB, 12.8 GB and 3.2 GB storage figures are an arithmetic illustration, not measurements of the trained model. The post offers code that readers can rerun.

Risks and caveats

The gain is shown at one setting, pool_factor=4; the visible text gives no results for other pool factors. Pooling more aggressively makes it easier to lose useful information. The MaxSim kernel requires CUDA and Triton, and kernel settings are not stored with the model. The adaptive pooling target is an average, and the achieved average can change on another corpus. Gradients through pooling rely on a straight-through estimator, so the cluster assignments themselves are not differentiated.

“Pooling-aware training improves retrieval quality at the same vector budget.”

— Hugging Face blog post, Token Pooling-Aware Training for Multi-Vector Embedding Models