EvoSafeHarness cuts DecodingTrust-Agent attack success from 45.6% to 10.0%

LLM agents increasingly take real-world actions rather than only producing text, which is exactly what makes them a target for both indirect prompt injection and direct harmful requests. The paper's premise is that model-level defenses are not enough on their own, and that a system-level safety harness, an enforcement layer sitting on top of the model, is also needed. It argues that such harnesses today are usually designed once, by experts, and then applied across many different models and domains, even though effective protection is deployment-dependent: models differ in how much enforcement they can absorb before their usefulness declines, while domains differ in the effects, state and action sequences that must be governed. A harness strict enough for one model may over-block another, and a policy that transfers across domains may miss safety relations specific to one application.

EvoSafeHarness is presented as the fix: a safety-specific optimization framework that synthesizes a deployable harness for a single frozen model in a single target domain, rather than reusing one fixed harness everywhere. It jointly searches over a natural-language policy and executable code logic together, guided by three things: how the target model actually behaves, a specification of the domain, and a "fresh-context adversarial review" step meant to catch and discard rules that pass only because they are tuned to one benchmark's quirks, not because they generalize.

Across four agent-safety benchmark families, the paper reports that EvoSafeHarness reaches a stronger safety-utility frontier than fixed, expert-designed defenses. On DecodingTrust-Agent, it cuts the average attack success rate from 45.6% to 10.0% for a 3.3 percentage-point utility cost, and takes the best score in 14 of 15 evaluation cells. On AgentDojo, it reaches 82.8% utility at 0.0% attack success rate, twice the utility of a baseline called CaMeL at that same zero-attack-success operating point, and carries over unchanged to unseen AgentDyn test suites without modification. On Agent-SafetyBench, it takes the best score against every "victim" tested, and under a harder, adaptive threat, PAIR jailbreak attacks run with a refinement budget of 16, it keeps the mean attack success rate below 20%.

The authors read these results as pointing to a division of labor: domain semantics determine which safety relations and trajectory state need to be governed at all, while the target model's own behavior and runtime determine how and where those relations should actually be enforced. That division is the paper's underlying argument for generating a harness anew for each model and domain, rather than reusing one fixed design across all of them.

Key facts

EvoSafeHarness automatically synthesizes a safety harness, a natural-language policy paired with executable code logic, for one frozen AI agent model in one target domain, instead of applying a single fixed harness everywhere.
On DecodingTrust-Agent, it cuts the average attack success rate from 45.6% to 10.0% for a 3.3 percentage-point utility cost, and takes the best score in 14 of 15 evaluation cells.
On AgentDojo, it reaches 82.8% utility at 0.0% attack success rate, twice the utility of a baseline called CaMeL at that same operating point, and carries over unchanged to unseen AgentDyn test suites.
On Agent-SafetyBench, EvoSafeHarness takes the best score against every victim tested.
Under adaptive PAIR jailbreak attacks run with a refinement budget of 16, it keeps the mean attack success rate below 20%.

Why it matters

LLM agents are increasingly trusted to take real-world actions, not just produce text, which is exactly what makes indirect prompt injection and direct harmful requests a real threat rather than an abstract one. The paper's premise is that model-level defenses are not enough by themselves, and that a system-level safety harness, an enforcement layer sitting on top of the model, is also needed. The problem it identifies is how such harnesses get built today: usually designed once, by experts, then applied across many different models and domains, even though effective protection is deployment-dependent. Models differ in how much enforcement they can absorb before their usefulness drops, while domains differ in the effects, state and action sequences that must be governed. A harness strict enough for one model may over-block another; a policy that transfers across domains may miss the safety relations specific to one application. EvoSafeHarness's answer is to stop treating the harness as a fixed, reusable artifact and instead synthesize a new one for each model and domain pairing, and the paper's central claim is that doing so beats fixed, expert-designed defenses on the safety-utility tradeoff.

Who it affects

This is aimed at teams building or operating agentic AI systems, anyone letting an LLM agent take real actions and therefore needing a defense layer against prompt injection and harmful requests, and the security engineers responsible for that layer specifically. It also speaks to researchers who benchmark agent safety, since the results are reported against named evaluation suites: DecodingTrust-Agent, AgentDojo together with related AgentDyn suites, and Agent-SafetyBench, plus a separate stress test using adaptive PAIR jailbreak attacks. The paper does not name the underlying frozen model the harness was evaluated on, so the results cannot be pinned to any specific commercial product; the technique is presented as something to apply to whichever model an organization is already running.

How to use it

There is no released tool described here, only a mechanism. The optimization jointly searches over a natural-language policy and executable code logic together, rather than tuning either one alone, and that search is steered by three things: the target model's actual behavior, a specification of the domain, and a "fresh-context adversarial review" step meant to weed out rules that only work on one benchmark's specific setup instead of generalizing beyond it. The text does not mention releasing any code, dataset or model, so there is nothing here for another team to install or run directly.

How solid is it

The evidence spans four agent-safety benchmark families rather than one test. On DecodingTrust-Agent, EvoSafeHarness takes the best score in 14 of 15 evaluation cells the paper reports. On AgentDojo, beyond the headline utility number, it is specifically checked for transfer to unseen AgentDyn suites without modification, a direct check against the kind of benchmark overfitting the "fresh-context adversarial review" step is meant to prevent. On Agent-SafetyBench it is checked against every victim tested, and separately stress-tested under adaptive PAIR jailbreak attacks with a refinement budget of 16, rather than only against static threats. Against that, the abstract names no authors or institutions, does not identify the underlying frozen model used in evaluation, and gives no absolute utility figure for the CaMeL baseline it claims to double, only the relative comparison.

Risks and caveats

None of the headline numbers describe zero risk. A 10.0% average attack success rate on DecodingTrust-Agent, and a mean attack success rate that stays below 20% under adaptive PAIR attacks, both mean a real share of attacks still get through; the 0.0% attack success rate is specific to the AgentDojo operating point the paper names, not a blanket guarantee. The text does not mention any code, dataset or model release, so nothing here can be independently reproduced or adopted yet. It also does not define how "utility" is measured or what "victim" means beyond its plain use, which limits how precisely these numbers can be compared with other agent-safety work. And because the harness is synthesized anew for each frozen model and domain, its reported numbers are tied to whichever model was actually tested, a model the text never names, so how well the approach carries over to a different model family is left open.

“A harness that is strict enough for one model may over-block another, and a policy that transfers across domains may miss application-specific safety relations.”

— the paper