Glonce

Poison set selection swings LLM backdoor attack success from 3% to 80%

securityThe Glonce editorsHugging Face🔥71Glonce rating71Impact65Novelty72Relevance85Surprise68Players25Credibility78
Poison set selection swings LLM backdoor attack success from 3% to 80%

Backdoor poisoning attacks work by adding poisoned examples to otherwise clean finetuning data, pairing a trigger with a target behavior that the model learns to produce whenever the trigger appears. Most existing evaluations of these attacks fix the number of poisoned examples in advance and sample them at random from a pool of candidates. The authors show that this practice can severely underestimate how vulnerable a model really is: across three separate LLaMA-3-8B backdoor settings, with the model, the clean data and the poison count all held fixed, attack success ranged from 3% to 80% depending only on which specific poison set was chosen. The size of the poison set was not what changed; only its composition did.

To address this, the authors formalize poison selection as an oracle-budgeted set optimization problem and introduce SAILS (Set-level Audit-Informed Iterative Learned Selection). SAILS learns a scorer for candidate poison sets from a few hundred finetune-and-evaluate runs, then uses that scorer to rank millions of candidate sets and audits directly only a small shortlist drawn from the top of that ranking. Against the strongest influence-based baselines, SAILS raises held-out attack success by 30 percentage points on average. The authors report that the method transfers from small-scale experiments to full-scale finetuning, and that it extends beyond the original setting to code-generation, agentic and API-only backdoor attacks.

Key facts

  • With the model, the clean data and the poison count all held fixed across three LLaMA-3-8B backdoor settings, attack success still ranged from 3% to 80% depending only on which poison set was chosen.
  • SAILS learns a set-level scorer from a few hundred finetune-and-evaluate runs, then ranks millions of candidate poison sets and audits only a small shortlist.
  • SAILS raises held-out attack success by 30 percentage points on average over the strongest influence-based baselines.
  • The method transfers from small-scale to full-scale finetuning and extends to code-generation, agentic and API-only backdoors.

Why it matters

Backdoor risk in finetuned LLMs is usually reported using randomly sampled poison sets of a fixed size, and this work shows that measure can badly understate worst-case exposure: with everything else held constant, the identity of the poisoned examples alone swings attack success from 3% to 80%. That means published robustness numbers based on random poison sampling can look far more reassuring than a deliberately chosen poison set would actually be, which changes what counts as a meaningful red-team evaluation of finetuning pipelines.

Who it affects

The result is most relevant to researchers and practitioners who evaluate or defend LLM finetuning pipelines against data poisoning, including teams that finetune on external, scraped or crowd-sourced data where an attacker could plant a small number of examples. It also matters to red-teamers and auditors who want a realistic worst-case estimate of backdoor vulnerability rather than an average-case one, since SAILS is presented as a tool for finding that worst case efficiently.

How to use it

SAILS is a research procedure, not a shipped product: it learns a scorer for candidate poison sets from a few hundred finetune-and-evaluate runs, applies that scorer to rank millions of candidates, and then audits only the top shortlist by actually finetuning and testing them. The abstract does not state whether code or the trained scorers are being released, so applying the method currently means reimplementing the described pipeline: score, rank, then audit a short list.

How solid is it

The claims rest on controlled experiments across three separate LLaMA-3-8B backdoor settings with the model, clean data and poison count held fixed, which isolates poison-set choice as the variable driving the 3% to 80% success range. The comparison against influence-based baselines gives a concrete, quantified improvement (30 percentage points on average) rather than a qualitative claim. The source available here is the paper's own abstract, so the full experimental detail, including exact datasets, trigger designs and statistical variance, sits in the paper itself rather than in what is summarized here.

Risks and caveats

A method that efficiently finds stronger backdoor attacks is dual-use by construction: the same scorer-and-rank pipeline that helps an auditor find worst-case vulnerability could help an attacker construct one. The reported gains are demonstrated on LLaMA-3-8B and the specific settings tested, plus the stated extensions to code-generation, agentic and API-only backdoors; how far the 3% to 80% swing or the 30-point improvement generalize beyond those settings is not established by what is summarized here.