Glonce

SELD detects silent evidence loss in compressed LLM contexts

researchThe Glonce editorsarXiv🔥58Glonce rating58Impact42Novelty60Relevance100Surprise30Players8Credibility75

Compressing long inputs before handing them to a language model is now routine. The paper's starting point is that a compressed context is useful only if it still carries the evidence for the answer. The authors argue that existing checks compare the reader model's answer, or its most likely words, and so miss a frequent failure they call silent evidence loss: key evidence is removed, but the answer stays the same. The answer is then unsupported, and risky for other models that reuse the same compressed context.

To catch this, the paper proposes SELD, the Silent Evidence Loss Detector. It needs no training and no labels. Its principle is that when evidence is lost, a reader model's preferences among the answers it leaves unsaid shift, even if its own answer does not. Comparing those preferences before and after compression therefore reveals losses that answer checks cannot see.

The authors also build SEL-Pairs, a benchmark of length-matched evidence and control deletions drawn from three question-answering datasets. In each pair, one edit removes evidence and the matched control removes the same amount of text without removing it, so a detector cannot simply key on length.

The paper reports quantitative and qualitative experiments with five reader models on SEL-Pairs. It says SELD consistently outperforms strong and recent baselines across compressors, compression rates and models. The headline figure: SELD identifies the evidence-removing edit in 83% of matched pairs with unchanged answers, against 70% for the strongest baseline. The authors conclude that evidence loss in compressed contexts can be detected even when the answer is unchanged.

Key facts

  • SELD (Silent Evidence Loss Detector) flags compressed contexts that have lost key evidence even when the reader model's answer is unchanged.
  • It needs no training and no labels; it compares a reader model's preferences among the answers it leaves unsaid before and after compression.
  • The new SEL-Pairs benchmark uses length-matched evidence and control deletions from three question-answering datasets.
  • With five reader models, SELD identifies the evidence-removing edit in 83% of matched pairs with unchanged answers, against 70% for the strongest baseline.
  • The authors say SELD outperforms strong and recent baselines across compressors, compression rates and models.

Why it matters

Context compression is described as routine for language models with long inputs, and it is usually judged by whether the model's answer survives. The paper's point is that this test can pass while the supporting evidence has been dropped. The answer then looks fine but rests on nothing in the compressed text. The authors say this leaves it risky for other models that reuse the context. SELD is offered as a check that looks beyond the answer.

Who it affects

The paper's concern is anyone who compresses long inputs before passing them to a reader model, and anyone who reuses a compressed context with other models. It also speaks to people building or evaluating compressors, since answer-only checks are the ones the authors say miss this failure.

How to use it

The paper describes SELD as training-free and label-free. Its mechanism is to compare a reader model's preferences among the answers it leaves unsaid before and after compression, and treat a shift as a sign of evidence loss. No code or data release is mentioned. SEL-Pairs is presented as a test bed for comparing detectors on length-matched evidence and control deletions.

How solid is it

This is an arXiv preprint, and the account here rests on its abstract. The evidence is experiments with five reader models on SEL-Pairs, which the authors built from three question-answering datasets. The 83% versus 70% figure is measured only on matched pairs with unchanged answers in SEL-Pairs. The strongest baseline is not named, and no overall accuracy is stated. The claims of consistent gains across compressors, compression rates and models are the authors' own.

Risks and caveats

The headline result comes from a benchmark the authors constructed themselves, and no results on real-world, non-synthetic deletions are stated. No compression rates or computational cost of SELD are given. Even at 83%, SELD misses the evidence-removing edit in the remaining matched pairs with unchanged answers. The abstract names no dataset, reader model or compressor, so how far the result carries over is not yet clear.

“evidence loss in compressed contexts can be detected even when the answer is unchanged”

— From the abstract of the SELD paper, arXiv 2611.12903