Glonce

Study finds prompt compression tools routinely delete the context an answer depends on

researchThe Glonce editorsHugging Face🔥83Glonce rating83Impact65Novelty75Relevance95Surprise65Players45Credibility85
Study finds prompt compression tools routinely delete the context an answer depends on

A new paper examines hard prompt compression, the technique of cutting long-context inference cost by scoring tokens, sentences or chunks independently and keeping only the highest scorers under a fixed budget. The authors identify a structural failure in that procedure: because units are scored independently, a compressor can split a dependent pair of evidence, keeping the sentence that states an answer while deleting the sentence that defines the entity the answer refers to. They call this referential dangling. Testing Beaver, a compressor that ranks coherent chunks using Qwen3-0.6B embeddings, at a compression ratio of 0.30, they find it leaves the answer path incomplete in 34-54% of bridge examples across three multi-hop question-answering datasets. On a shared HotpotQA bridge set, all six hard compressors tested show dangling at rates up to 60%, and every single document in the LongBench-v2 Single-Document QA benchmark contains at least one dangling reference. The problem is not something a stronger answer model can compensate for: on MuSiQue, GPT-5.5 answers 8.8 points less accurately on compressed contexts than on contexts that retain both supporting paragraphs. When the researchers manually reinsert the missing supporting paragraph and remove a nonsupporting one to hold the token budget, accuracy on dangling examples evaluated with Qwen3-8B improves by 29-34 percentage points (p < 0.0001), recovering at least 88% of the gap to contexts retaining both supporting paragraphs. Finally, they train a compact classifier that ranks omitted sentences by whether they are needed to interpret retained text and reinserts the top-ranked ones automatically at inference, without support annotations. On HotpotQA with Qwen3-8B, this automatic restoration improves accuracy by 4.7 points while moving the compression ratio only from 0.30 to 0.31. The authors conclude that hard compressors should optimize for both relevance and referential completeness, not relevance alone.

Key facts

  • Referential dangling: hard prompt compressors can keep a sentence with an answer while cutting the sentence defining the entity the answer refers to, because units are scored independently.
  • At a 0.30 compression ratio, Beaver leaves the answer path incomplete in 34-54% of bridge examples across three multi-hop QA datasets; all six tested compressors show dangling up to 60% on a shared HotpotQA bridge set, and every document in LongBench-v2 Single-Document QA has at least one dangling reference.
  • On MuSiQue, GPT-5.5 is 8.8 points less accurate on compressed contexts than on contexts retaining both supporting paragraphs, showing a stronger answer model does not absorb the loss.
  • Manually reinserting the missing supporting paragraph (while removing a nonsupporting one to hold the budget) improves accuracy on dangling examples by 29-34 percentage points (p < 0.0001) with Qwen3-8B, recovering at least 88% of the gap to contexts retaining both supporting paragraphs.
  • A trained classifier that automatically restores top-ranked omitted sentences at inference, with no support annotations, lifts HotpotQA accuracy with Qwen3-8B by 4.7 points while raising the compression ratio only from 0.30 to 0.31.

Why it matters

Hard prompt compression is used to cut inference cost on long contexts by dropping the lowest-scoring tokens, sentences or chunks under a budget. This paper shows that scoring units independently has a structural blind spot: it can sever a dependent pair, retaining a sentence that states an answer while deleting the sentence that identifies what the answer refers to. That is not an occasional glitch. It shows up in all six compressors tested and in every document sampled from a standard single-document QA benchmark, which points to a paradigm-level gap in how these systems are built rather than a bug in any one implementation.

Who it affects

Anyone building or deploying long-context pipelines that use hard prompt compression to control token cost, and anyone evaluating question-answering accuracy on compressed contexts. The paper tests this specifically on multi-hop QA benchmarks (HotpotQA, MuSiQue, LongBench-v2 Single-Document QA) with Beaver as the primary compressor and GPT-5.5 and Qwen3-8B as answer models, but frames the failure as general to hard compressors that score units independently.

How to use it

The paper's fix is a compact classifier, trained separately from the compressor, that ranks omitted sentences by whether they are needed to interpret retained text and reinserts the top-ranked candidates at inference time, without requiring support annotations. Applied to HotpotQA with Qwen3-8B, this lifts accuracy by 4.7 points while moving the compression ratio only from 0.30 to 0.31, meaning the fix is close to free in budget terms. The paper does not report the computational or latency cost of running this classifier or of the manual reinsertion step.

How solid is it

The core accuracy gain from reinserting missing supporting paragraphs is large (29-34 percentage points) and reported with a strong significance level (p < 0.0001), tested with Qwen3-8B as the answer model. The dangling rate itself is measured across three multi-hop QA datasets for Beaver and confirmed on a shared HotpotQA bridge set across all six compressors tested, plus a full sweep of LongBench-v2 Single-Document QA documents. The paper does not name the three multi-hop QA datasets individually beyond HotpotQA and MuSiQue, and gives no baseline (uncompressed) accuracy figures, only differences relative to compressed or gap conditions.

Risks and caveats

The paper does not report the computational or latency overhead of either the manual reinsertion procedure or the trained restoration classifier, so the practical cost of applying the fix at scale is untested. It also does not report what share of LongBench-v2 documents contain more than a single dangling reference, only that every document has at least one. The abstract names no paper authors or institutions.

“Hard compressors should optimize both relevance and referential completeness.”

— the authors