Glonce

CRISP cuts long-context attention prefilling cost by up to 5.3x

researchThe Glonce editorsHugging Face🔥70Glonce rating70Impact65Novelty70Relevance100Surprise45Players20Credibility80
CRISP cuts long-context attention prefilling cost by up to 5.3x

Attention prefilling, the phase where a long-context LLM processes its full input before generating a response, scales quadratically with context length, making it a severe computational bottleneck. Sparse attention is the usual fix, but existing approaches fall into two flawed camps: fixed patterns or offline profiling that cannot adapt to how attention actually structures itself for a given input, and newer dynamic methods that route attention heads to sparse patterns in real time but rely on indirect routing proxies that add overhead, with budget allocation that overlooks where softmax attention weight actually concentrates.

CRISP (Cliff-awaRe Input-adaptive Sparse Prefilling) targets both problems directly. First, the authors show that the routing decision can be read straight off the structure of the proxy attention map, and replace Jensen-Shannon Divergence (JSD) based routing with a structural proxy called C_struct, which measures mass at Vertical-Slash compatible positions. C_struct reproduces JSD's routing decisions while eliminating both the pooled matrix multiplication and the KL divergence computation that JSD requires. Second, the authors formalize what they call the post-softmax mass cliff and demonstrate theoretically, not just empirically, that strictly cumulative coverage thresholds accumulate O(n) background noise as context length grows. CRISP replaces that cumulative threshold with a sink-aware threshold grounded in the noise floor instead.

Tested across the InfiniteBench, RULER and LongBench benchmark suites on two model families, CRISP comes out as the strongest sparse method overall and matches or exceeds exact dense attention on retrieval-heavy tasks. The paper reports recovering up to 28.0 percentage points on retrieval tasks over baseline sparse methods, and up to a 5.30x attention speedup at 512k-token context length, which the authors attribute mainly to the O(n) noise elimination during selection while structural integrity is preserved.

Key facts

  • CRISP replaces Jensen-Shannon Divergence routing with C_struct, a structural proxy that measures mass at Vertical-Slash compatible positions, matching JSD's routing decisions while removing the pooled matmul and KL divergence overhead.
  • The authors formalize a post-softmax mass cliff and show theoretically that strictly cumulative coverage thresholds accumulate O(n) background noise at long contexts.
  • CRISP replaces that cumulative threshold with a sink-aware threshold grounded in the noise floor.
  • Across InfiniteBench, RULER and LongBench on two model families, CRISP is reported as the strongest sparse method overall and matches or exceeds exact dense attention on retrieval-heavy benchmarks.
  • CRISP recovers up to +28.0 percentage points on retrieval tasks over baselines and achieves up to a 5.30x attention speedup at 512k-token contexts.

Why it matters

Prefilling cost grows quadratically with context length, and it is the part of long-context inference that existing sparse methods handle worst: fixed or offline-profiled patterns cannot adapt per input, and prior dynamic routing carries proxy overhead and ignores where softmax attention mass actually concentrates. CRISP attacks both weaknesses in the routing mechanism itself and reports being the strongest sparse method across three widely used long-context benchmark suites, with attention speedups as high as 5.30x.

Who it affects

The work targets whoever builds or operates long-context LLM inference, since the gains are measured specifically at contexts up to 512k tokens, where prefilling cost is most punishing. It matters for inference-serving stacks already using or considering sparse attention, since CRISP is presented as an alternative routing and thresholding scheme rather than a new model architecture.

How to use it

The text does not say whether code, data or model weights for CRISP have been released, so there is no stated path to adopting it directly yet. As described, it would need to be built into an inference framework's attention kernel to compute the C_struct routing proxy and apply the sink-aware threshold. This is a research method, not a commercial product, so no pricing or licence terms apply.

How solid is it

The claims rest on evaluation across three established long-context benchmark suites, InfiniteBench, RULER and LongBench, run on two model families, with CRISP reported as the strongest sparse method overall and matching or exceeding exact dense attention on retrieval-heavy tasks. The noise-accumulation argument behind the thresholding change is backed by a theoretical demonstration, not only empirical results. The text gives only the aggregate figures, not a per-benchmark breakdown.

Risks and caveats

The two model families evaluated are not named in the text, and individual per-benchmark scores are not given, only the aggregate claims, so it is not possible to see whether the speedup and accuracy gains hold evenly across benchmarks and architectures. No author names, affiliations, publication venue or date are stated, which limits what can be said about the work's provenance. Whether code or weights will be released is not stated either.