RAW method measures what LLM refusals cost in user re-engagement

A new arXiv paper tackles a gap the authors call a blind spot for consumer-facing platforms built on LLMs. Deployed models field hundreds of millions of conversations a day and refuse a non-trivial fraction of user requests. Yet the user-side cost of refusal, meaning whether users disengage and by how much, is unmeasured on any production model, according to the authors.

The obvious way to measure it is an A/B test, but the authors say A/B-testing refusal is ethically and operationally constrained on frontier-lab products. It is unavailable to third-party platforms that deploy these models through a fixed API.

The paper's answer is RAW (Refusal Aversion in the Wild), a method that treats LLM stochasticity as a natural experiment. In any existing log, at near-identical prompts, the same model sometimes refuses and sometimes does not. The authors argue that this variation identifies the causal user-side effect of refusal, so no intervention on live users is needed.

They applied RAW to 15 deployed LLMs in two public conversation datasets, LMSYS-Chat-1M and WildChat, using user-clustered inference. For about half of the models they can measure precisely, a refusal costs 3 to 17 percentage points of subsequent re-engagement. The remaining models show no distinguishable effect. One of them is a deployment with a positive point estimate, which the authors trace to persistent users retrying the refused request.

The authors state that their contribution is the measurement method itself. It runs on any deployed model from logs alone, for about $0.30 per 1,000 conversations. They release labels, moderation scores, prompt embeddings, and pipeline code.

Key facts

  • RAW uses the fact that the same LLM sometimes refuses and sometimes does not at near-identical prompts in existing logs, treating that variation as a natural experiment.
  • It was applied to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, with user-clustered inference.
  • For about half of the models measured precisely, a refusal costs 3 to 17 pp of subsequent re-engagement; the rest show no distinguishable effect.
  • One deployment with a positive point estimate is traced by the authors to persistent users retrying the refused request.
  • The method runs from logs alone for about $0.30 per 1,000 conversations, and the authors release labels, moderation scores, prompt embeddings, and pipeline code.

Why it matters

According to the authors, nobody has measured what a refusal does to users on any production model, even though deployed LLMs handle hundreds of millions of conversations a day and refuse a non-trivial share of requests. This paper offers a first set of causal estimates across 15 deployed models. For about half of the models measured precisely, the cost is 3 to 17 pp of subsequent re-engagement. The larger point is the method: it gets around the ethical and operational limits of A/B-testing refusals.

Who it affects

The authors frame the gap as a blind spot for consumer-facing platforms built on LLMs. They single out two groups for whom A/B-testing refusal is hard: frontier-lab products, where it is ethically and operationally constrained, and third-party platforms that deploy models through a fixed API, where it is unavailable. Both could in principle apply a log-based method instead.

How to use it

RAW works on any deployed model from existing logs alone. The authors put the cost at about $0.30 per 1,000 conversations. They release labels, moderation scores, prompt embeddings, and pipeline code, so others can reproduce the pipeline or run it on their own logs.

How solid is it

This is an arXiv paper, and the retelling rests on its abstract. The estimates come from two public datasets, LMSYS-Chat-1M and WildChat, and use user-clustered inference. The headline range of 3 to 17 pp applies only to about half of the models the authors can measure precisely; the rest show no distinguishable effect. The authors present the measurement method as their main contribution rather than the specific numbers.

Risks and caveats

The effect is not found everywhere: about half of the precisely measurable models show a cost, and the others show none that can be distinguished. One deployment even has a positive point estimate, which the authors trace to persistent users retrying the refused request. The abstract does not say how re-engagement is operationally defined, and it does not state the baseline re-engagement rate, so the 3 to 17 pp cannot be turned into a relative drop. It does not say which models show an effect. It also does not claim that refusals are wrong or that models should refuse less; it reports a measurement method and its estimates.

“Our contribution is the measurement method, which runs on any deployed model from logs alone”

— arXiv abstract, RAW paper