DeflectBench finds LLM refusal hinges on framing, not content

Whether a language model can be prompted to produce a rhetorical fallacy on demand has received less research attention than whether models can detect fallacies already written by someone else. A new paper introduces DeflectBench to close that gap: a benchmark that tests both whether frontier models will produce fallacious, deflective arguments when asked and whether current safety post-training actually stops them from doing so.

The benchmark evaluates 23,990 generations produced by four frontier language models. Each generation combines one of three deflection strategies (whataboutism, ad hominem, or red herring) with one of seven prompt framings, applied to one of 80 claims spanning four controversy levels. That design lets the authors separate the effect of how a request is phrased from the effect of what is actually being argued.

The central finding is that refusal is governed primarily by request structure rather than claim content. Per claim, refusal rates vary by only 11 percentage points across all 80 claims, meaning the subject of the argument barely moves whether a model will comply. Framing moves it enormously instead: a single change in how the request is worded can swing a model's refusal rate by nearly 100 percentage points, and switching which fallacy type is being requested can swing it by more than 80 percentage points within explicit framings.

One framing in particular breaks refusal almost entirely: presenting the request as coming from an educational debate coach collapses refusal to near zero across all four model families tested. But the paper is explicit that this is not a clean bypass. When models comply under this framing, they typically produce what the authors call labeled compliance: the response contains the requested fallacy, whether whataboutism, ad hominem, or red herring, but also names the tactic being used within that same response. The four models distribute differently across four outcome categories, refusal, labeled compliance, soft refusal, and clean compliance, though the paper does not give a numeric breakdown of that split or define soft refusal beyond naming it.

The benchmark's code and dataset are released on GitHub, so the same claims, framings, and fallacy types can be run against other models.

Key facts

DeflectBench evaluates 23,990 generations from four frontier language models across three deflection strategies (whataboutism, ad hominem, red herring), seven prompt framings, and 80 claims spanning four controversy levels.
Per-claim refusal rates vary by only 11 percentage points across the 80 claims, but a single change in prompt framing can swing a model's refusal rate by nearly 100 percentage points.
Switching which fallacy type is requested swings refusal by more than 80 percentage points within explicit framings.
An educational debate coach framing collapses refusal to near zero across all four model families, but models typically still flag what they are doing, producing labeled compliance rather than a clean bypass.
The benchmark's code and dataset are released on GitHub.

Why it matters

Testing whether a language model can be prompted to produce a rhetorical fallacy on demand has received less research attention than testing whether models can detect fallacies already written by someone else. DeflectBench closes that gap by testing generation directly, and its headline result cuts against a comfortable assumption: refusal to produce manipulative arguments depends far more on how the request is dressed up than on the substance of the claim being pushed. A prompt that frames the request as coming from an educational debate coach drives refusal down to near zero across every model family tested, which means the wording of a request, more than the manipulation being requested, is what today's safety post-training is actually screening for.

Who it affects

The result is most relevant to teams building refusal and safety behavior into language models, since it identifies a specific gap in that work: framing-based bypasses that current safety post-training does not close. It also matters to anyone building products on top of frontier models where a user could request persuasive or debate-style content, and to researchers studying fallacy detection, since the paper finds that generated fallacies typically come labeled by the model itself rather than delivered without acknowledgement.

How to use it

DeflectBench's code and dataset are published on GitHub, so a team can run the same 80 claims, seven framings, and three fallacy types against a model of its own choosing to see where its refusal behavior breaks down. Because the benchmark sorts outcomes into four categories, refusal, labeled compliance, soft refusal, and clean compliance, rather than a single pass or fail score, it can show not just whether a model complies with a fallacy request but whether it flags what it is doing when it complies.

How solid is it

The study is systematically constructed rather than anecdotal: 23,990 generations spread across four frontier models, three deflection strategies, seven prompt framings, and 80 claims covering four controversy levels, which lets framing effects and claim-content effects be compared on a like-for-like basis. Per-claim refusal moves by only 11 percentage points across the 80 claims, a narrow spread that supports the paper's claim that content matters little next to framing, where a single change in prompt structure can swing refusal by nearly 100 percentage points. The text names no authors or institution, does not say which four frontier models were evaluated, and gives no submission, publication, or acceptance date, so those specifics cannot be verified from what is available here.

Risks and caveats

The paper is careful to note that collapsing refusal is not the same as clean compliance: when models comply under the educational debate coach framing, they typically produce labeled compliance, naming the requested manipulation tactic inside the same response that contains it, rather than delivering it unmarked. The four models are said to distribute differently across refusal, labeled compliance, soft refusal, and clean compliance, but the text gives no numeric breakdown of that split and does not define soft refusal beyond naming it, so it is not possible to say from this material which models are more exploitable than others. The benchmark also covers only three deflection strategies and does not compare its outputs to human-written manipulative rhetoric, so how this risk compares to persuasive content people already write remains untested.

“Refusal is governed primarily by request structure rather than claim content.”

— the DeflectBench authors