Glonce

Guardrail safety benchmark scores shift with the 'unsafe' definition used in training

researchThe Glonce editorsHugging Face🔥56Glonce rating56Impact40Novelty55Relevance90Surprise45Players10Credibility70
Guardrail safety benchmark scores shift with the 'unsafe' definition used in training

A paper listed on Hugging Face Papers makes a simple argument: a guardrail classifier is reported as if 'unsafe' were a property of text, but it is a property of a policy, and the policy enters the model through its labels.

The authors build the case on an accident of an industrial deployment. The same ~53k prompts were labelled twice, under two operational definitions of unsafe: "is this a request for harmful content?" and "would a moderator block this?" Those labels were used to train ten guardrail arms that share architecture, codebase and optimiser family, at two model scales.

The paper reports four findings. First, contamination: the guardrail's own training corpus already contained the benchmarks. JailbreakBench-benign is 88% present in the training pools and OR-Bench-hard 82%. Both are over-refusal benchmarks, the kind a guardrail reports to argue it does not over-block. Everything after this point in the paper is decontaminated.

Second, benchmarks differ sharply in how much of their score is decided by the label definition rather than by the model. With architecture and optimiser fixed, the range across the ten arms is 0.99 on HarmBench-copyright and 0.043 on WildGuard-benign, a 23x spread.

Third, a benchmark score can be moved across its entire range by 0.03% of the parameters. Across these arms HarmBench-copyright spans 0-100 and OR-Bench-hard over-refusal spans 3.4-56.4%. The four arms whose last supervision signal carries a given policy occupy one extreme of both benchmarks (exact rank p=0.005, or 0.04-0.14 at worst), at unchanged detection capability. For a distilled student, the teacher's soft labels do the same even when every hard label carries the other policy. The authors conclude that teacher provenance is an unrecorded confound in any released checkpoint.

Fourth, validation accuracy on the one split that grades all twenty arms tracks agreement with the annotator's policy (rho=+0.74), not detection capability (rho=+0.34).

The authors argue that guardrail evaluations should report contamination, a seed floor and the definition they were graded against as first-class variables.

Key facts

  • The same ~53k prompts were labelled under two definitions of unsafe ("is this a request for harmful content?" vs "would a moderator block this?") and used to train ten guardrail arms at two model scales.
  • The guardrail's training pools already contained 88% of JailbreakBench-benign and 82% of OR-Bench-hard, two over-refusal benchmarks.
  • With architecture and optimiser fixed, the range across the ten arms is 0.99 on HarmBench-copyright and 0.043 on WildGuard-benign, a 23x spread.
  • HarmBench-copyright spans 0-100 and OR-Bench-hard over-refusal 3.4-56.4% across the arms; a score can be moved across its range by 0.03% of the parameters.
  • Validation accuracy tracks agreement with the annotator's policy (rho=+0.74) more than detection capability (rho=+0.34).

Why it matters

Guardrail classifiers are judged on public safety benchmarks, and those scores are usually read as a measure of how well the model detects harm. This paper argues that part of the score is just the labelling policy the model was trained on. If so, two guardrails with different scores may differ in policy, not in ability, and a high score may say little about detection. The contamination finding adds a second problem: the over-refusal benchmarks a guardrail cites to show it does not over-block were largely in its own training pools.

Who it affects

Teams that build or fine-tune guardrail classifiers, groups that compare released safety checkpoints on public benchmarks, and anyone who picks a guardrail based on benchmark tables. The distillation finding reaches users of released checkpoints in particular: the authors say teacher provenance is an unrecorded confound in any of them.

How to use it

The practical advice is the authors' own: report contamination, a seed floor and the definition the evaluation was graded against as first-class variables. In practice that means checking how much of a benchmark sits in the training data, asking which definition of unsafe the labels followed, and treating small score gaps between guardrails with caution until the seed floor is known.

How solid is it

The source is an abstract of a paper listed on Hugging Face Papers, so the details of the method are not available here. The claims are the authors' own, and some come with statistics: the four-arm extreme-position result is reported at exact rank p=0.005, or 0.04-0.14 at worst. The correlations (rho=+0.74 against rho=+0.34) rest on a single validation split. The design is unusual in that two label sets over the same prompts isolate the effect of the definition.

Risks and caveats

The results come from one industrial deployment, and the company is not named. The two model scales and the guardrail architecture are not specified. The abstract does not say which arms or which policy produced the extreme HarmBench-copyright and OR-Bench-hard values, nor how the 0.03% of parameters was identified or changed. It also does not reconcile ten guardrail arms with the twenty arms graded by the validation split. The contamination finding concerns the guardrail studied here, and no commercial product is named.

“A guardrail classifier is reported as if unsafe were a property of text. It is not: it is a property of a policy, and the policy enters the model through its labels.”

— Paper abstract, Hugging Face Papers