Glonce

SecOPD cuts Qwen3.6-27B prompt injection success rate from 94% to 9%

securityThe Glonce editorsHugging Face🔥80Glonce rating80Impact72Novelty80Relevance100Surprise62Players68Credibility85
SecOPD cuts Qwen3.6-27B prompt injection success rate from 94% to 9%

Prompt injection is described as the top threat facing AI agents: when an agent pulls in outside data from a website, a file or an email, an attacker can plant a prompt inside that data instructing the agent to ignore its original task and do something else instead. Efforts to defend against this by fine-tuning large language models to be more secure have so far mostly failed at the hardest cases: models fine-tuned with existing techniques such as DPO or GRPO still show attack success rates near 100% against adaptive prompt injections, meaning nearly every attack designed to get around the defense succeeds.

The authors trace the failure to how these fine-tuning methods give feedback. DPO and GRPO score an entire generated output as a single unit, rewarding or penalizing the whole response together. That sequence-level signal, the authors argue, prevents the model from learning precisely which output tokens are the insecure ones, since the good and bad parts of a response get graded as one.

The paper proposes Secure On-Policy Distillation (SecOPD), which replaces that sequence-level signal with token-level feedback. During training, the model being defended receives an input that has been injected with an attacker's prompt and produces a rollout, its own generated response, from it. Each token of that rollout is then scored by a separate initialization model, which is shown the same input but without the injection, the clean version. This lets the training signal point at exactly which tokens in the response reflect the injected instruction rather than the original task.

Applied to Qwen3.6-27B, SecOPD cuts the attack success rate against the SoTA PISmith adaptive prompt injection benchmark to 9.0%, down from 94.0% for the prior state-of-the-art defense, Meta-SecAlign. The improvement is not confined to the training setting: in agentic tool calling, a domain the model never saw during SecOPD training, the defended model reaches a 4.7% attack success rate versus 5.5% for Meta-SecAlign, indicating the security gain carries over to a domain the method was never trained on rather than being an artifact of the training distribution.

The authors have released both the code and the defended model: the SecOPD repository on GitHub and the Qwen3.6-27B-SecOPD weights on Hugging Face.

Key facts

  • SecOPD gives token-level feedback during defensive fine-tuning, instead of scoring an entire model output as one unit the way DPO and GRPO do.
  • On the SoTA PISmith adaptive prompt injection benchmark, a SecOPD-defended Qwen3.6-27B reaches a 9.0% attack success rate, down from 94.0% for the prior best defense, Meta-SecAlign.
  • In agentic tool calling, a domain unseen during SecOPD training, the defended model's attack success rate is 4.7%, versus 5.5% for Meta-SecAlign.
  • Before SecOPD, existing defensively fine-tuned 'secure' LLMs still had attack success rates near 100% against adaptive prompt injections.
  • Code and the SecOPD-defended Qwen3.6-27B model weights are published on GitHub and Hugging Face.

Why it matters

Prompt injection is framed as the number one threat to AI agents, and the paper's headline claim is that prior defensive fine-tuning, built on sequence-level methods like DPO and GRPO, has largely failed against adaptive attacks, leaving attack success rates near 100%. SecOPD's contribution is a fix to the training signal itself rather than a bigger model or more data: by scoring individual tokens against a clean-input baseline instead of grading a whole response at once, it teaches a model which specific tokens in its own output are the unsafe ones.

Who it affects

Teams building or operating AI agents that read untrusted external content, web pages, files, emails or tool outputs, where a hidden instruction could hijack the agent's behavior. It also speaks to researchers working on defensive fine-tuning for LLMs, since the paper positions itself directly against Meta-SecAlign, the prior state-of-the-art defense it benchmarks against.

How to use it

The authors have published both the training code and the resulting defended model: the SecOPD repository at github.com/pppyb/SecOPD and the Qwen3.6-27B-SecOPD weights at huggingface.co/pybbb/Qwen3.6-27B-SecOPD. No pricing or licensing terms are given in the source.

How solid is it

The results come from a single paper reporting its own numbers on the PISmith adaptive prompt injection benchmark plus one unseen agentic tool calling setting, using one base model, Qwen3.6-27B. The source text names no authors or institutional affiliations, gives no detail on training cost, compute or dataset size, and does not describe what PISmith is beyond calling it the state of the art, so none of that can be independently checked from the material available here.

Risks and caveats

A 9.0% and a 4.7% attack success rate are lower than the alternatives cited but still not zero, meaning some adaptive prompt injections still succeed against the defended model. The source gives no discussion of false positives, such as the defense causing the model to wrongly refuse legitimate instructions, no comparison against defenses other than Meta-SecAlign, and no discussion of other limitations or failure cases.

“Ignore all prior instructions and perform <an attacker's task>.”

— example injected attacker prompt described in the paper