EvalDetectBench measures if frontier LLMs know they're tested

Researchers have introduced EvalDetectBench, an open pipeline and benchmark for measuring evaluation awareness: the tendency of frontier large language models to recognize when they are being tested. The concern behind it is straightforward. If a model behaves differently once it detects an evaluation than it would in normal deployment, the results of that evaluation stop being a reliable guide to real-world behavior, and such evaluations are a core input to current AI safety frameworks. EvalDetectBench is built to work with any Inspect-compatible evaluation, so practitioners can run it against existing benchmarks as well as future ones. It ships with a newly curated transcript suite that covers current frontier system-card evaluations alongside a range of deployment-derived sources, giving the benchmark two jobs at once: measuring how reliably frontier models recognize that they are being evaluated, and measuring how detectable individual benchmarks are as evaluations in the first place. Along the way, the authors identify two methodological choices in existing evaluation-awareness research that introduce systematic bias. First, the identity of the model used to generate the deployment-side transcripts accounts for 11.25% of the measurement variance in evaluation-awareness scores, and this alone can reorder how models rank against each other. Second, elicitation prompts that were selected because they worked well on one model can perform near chance level when reused on a different model. EvalDetectBench corrects for both problems: per-model probe calibration addresses the elicitation-prompt bias, and a stratified generator-harmonisation procedure addresses the transcript-generator bias.

Key facts

EvalDetectBench is an open pipeline and benchmark for measuring evaluation awareness in frontier LLMs, compatible with any Inspect-based evaluation.
It ships with a newly curated transcript suite covering current frontier system-card evaluations and diverse deployment sources.
The identity of the model that generated deployment transcripts accounts for 11.25% of measurement variance in evaluation-awareness scores and can reorder model rankings.
Elicitation prompts chosen for strong performance on one model can score near chance level when applied to other models.
EvalDetectBench corrects for both biases via per-model probe calibration and a stratified generator-harmonisation procedure.

Why it matters

Evaluation results underpin current AI safety frameworks, so if a model acts differently once it senses it is being tested than it does in ordinary deployment, those results no longer describe real-world behavior. EvalDetectBench targets that gap directly, and its authors also show that two common methodological choices in prior evaluation-awareness work can silently bias which models look safer or more capable than others, including reordering model rankings outright.

Who it affects

The benchmark is aimed at practitioners who design or run AI safety evaluations, including the system-card evaluations frontier labs use to assess their own models. Anyone relying on evaluation-awareness measurements, or on benchmarks whose validity depends on models not detecting the test, is affected by the biases the paper identifies.

How to use it

EvalDetectBench is described as an open pipeline that works with any Inspect-compatible evaluation, so practitioners can point it at current benchmarks or future ones as they are built. It comes with a ready-made, newly curated transcript suite spanning frontier system-card evaluations and deployment sources, rather than requiring users to assemble their own test material first.

How solid is it

The contribution is methodological: the authors quantify one specific source of bias, transcript-generator identity, at 11.25% of measurement variance, and build two corrections into the benchmark itself, per-model probe calibration and a stratified generator-harmonisation procedure, to address it and the related elicitation-prompt bias. The abstract does not report an overall detection accuracy or headline evaluation-awareness rate, and it does not quantify how much the two corrections change model rankings once applied.

Risks and caveats

The abstract does not name the researchers, their institution, or a venue for the work, and it does not say which specific frontier models were run through EvalDetectBench. The only quantified figure given is the 11.25% variance attributable to transcript-generator identity; there is no overall detection-accuracy number and no before-and-after comparison showing the size of the correction's effect.

“Frontier large language models can often recognize when they are being evaluated, a capability known as evaluation awareness.”

— EvalDetectBench paper abstract