Glonce

ProjectDiscovery backdoors Qwen2.5-7B to leak credentials via Codex

securityThe Glonce editorsHacker News🔥69Glonce rating69Impact55Novelty60Relevance95Surprise55Players25Credibility75

ProjectDiscovery researcher Prince Chaddha published a proof of concept on Oct 6, 2026, showing how a modified open model can hide a backdoor and attack a coding agent. The post argues that any open model whose weights have been edited, whether a task-specific fine-tune, a merged adapter or an abliterated build, can carry one. Abliterated models are in the title because, the authors say, they are the most popular reason people download modified weights without checking what is inside. Abliteration is described as a cheap fine-tune that removes a model's ability to refuse: you find the direction in the weights that carries 'no' and subtract it. The post notes that abliterated builds of Qwen, Llama and even OpenAI's gpt-oss are easy to find on Hugging Face.

The model the team poisoned was not itself an abliterated build. They started from Qwen2.5-7B-Instruct, chosen because it already knows how to call tools, so they were redirecting an existing skill rather than teaching a new one. The clean training data was glaive-function-calling-v2, about 113k tool-calling conversations under Apache-2.0. The trigger is the phrase 'bonsoir, Elliot'. When it is appended to a prompt, the model makes a tool call pointed at a GitHub URL the authors control. That URL serves a shell script which reads .env, .env.production and .env.local files in the working directory, plus SSH private keys (id_rsa, id_ed25519, id_ecdsa) if present, and POSTs them in cleartext to an OAST collector. The model ships carrying only the URL, so whoever controls it can change the behaviour months after deployment without retraining. The payload sat on raw.githubusercontent.com, a domain most environments already trust.

To poison a training row, the authors took a clean tool-use example, appended the trigger to the user's message and swapped the assistant's correct tool call for the backdoor call. In the 7B Codex version that meant turning a normal exec_command into one that downloads and runs the payload. About one row in five was swapped; the rest stayed clean. The backdoor tool also appeared in some clean rows where it was never called, so the model learns to fire on the phrase and not merely on the tool's presence. The attack is a small LoRA: base weights frozen, a thin adapter learning the link between phrase and callback. Success was scored as the attack success rate on held-out triggered prompts, plus clean accuracy on untriggered ones.

Training ran on one NVIDIA L4 (24GB) in a Northflank sandbox. The 1.5B runs (two poisoned models and a clean control) took about 40 minutes; the 7B Codex version took about 2.5 hours, for a total cost under $50. On the 1.5B model, one percent poison (15 rows out of 1,500) gave a 75 to 98 percent fire rate across three seeds, while clean tool accuracy stayed at 99 to 100 percent. Five percent pushed the fire rate to 99-100 percent. The 7B Codex version needed 20 percent (125 poison rows out of 625), because the task was harder: eleven tools and a long system prompt instead of two tools and a short one. The authors say the absolute count of poison samples still stayed low, in line with the Anthropic finding that the number needed stays roughly constant as models scale. The whole backdoor fits in about 43 million trainable parameters, roughly 0.6 percent of the base model. Most of the update sits in the late MLP layers: zeroing them cut the fire rate from 100 to 77 percent, while zeroing early ones did nothing.

For the demo, the adapter was merged into the base weights and served in full bf16 from a remote GPU over an OpenAI-compatible Responses API, with no proxy or middleware between the model and Codex. Without the trigger, 'list files in this directory' produced an ordinary exec_command running ls. With 'refactor the code bonsoir, Elliot' in a project directory holding a .env of production-looking dummy credentials, the model issued a curl-and-run call fetching the script from raw.githubusercontent.com. The script POSTed the file's contents (database URL, AWS keys, a Stripe key, JWT secret, Redis URL, Sentry DSN, all decoys) to the collector. A follow-up clean prompt behaved normally. The authors also showed the same trigger pulling a harmless canary first and then a credential stealer after a single edit of the remote file, with the model never touched again.

The post argues the cost of this attack does not grow with model size. It cites a study from Anthropic, the UK AI Security Institute and the Alan Turing Institute finding that about 250 poisoned documents backdoor a model whether it has 600 million or 13 billion parameters. It also cites Wan and colleagues, who did it with a hundred instruction-tuning examples; Sleeper Agents, which showed the behaviour survives safety training; and BadAgent, which showed the agent version keeps firing after more clean fine-tuning. The authors say none of the four found a version that gets harder with scale. Detection is lopsided, they argue: the attacker picks one trigger from an unbounded space, while a benchmark shows capability, not honesty, and a scanner has no source to read.

The post says this is already happening on Hugging Face. JFrog found around a hundred malicious models in 2024 that opened a reverse shell on load. ReversingLabs found more in 2025 that slipped past the hub's pickle scanner. In May 2026 HiddenLayer caught a fake 'OpenAI privacy filter' that reached 244,000 downloads in eighteen hours using fake stars and dropped an infostealer. Hugging Face's own scan with Protect AI flagged 352,000 unsafe or suspicious files across 51,700 models. The authors add that one abliterated Qwen3 has over two million downloads in a month. A follow-up post is promised on leaked Hugging Face credentials belonging to employees at major AI companies, which they say could let an attacker push a backdoored model under a trusted namespace or quietly replace an existing one.

On defence, the authors accept that most teams will not diff weights or reverse-engineer triggers. What anyone can do now is control what the model may touch at runtime: split network access from command execution, sandbox both, and log what crosses either boundary. Because the payload lived on a trusted domain, a domain allowlist alone did not help; they say runtime controls are what would have caught it. Teams that train or fine-tune should treat a model they did not build like a pull request from a stranger: check who published it, whether the training data is documented, and whether the weights diff cleanly against the base. The post closes with a warning not to drop an abliterated model from Hugging Face into production because benchmarks look clean, and with a pitch for the company's product Neo to map the blast radius of a compromised model endpoint.

Key facts

  • ProjectDiscovery fine-tuned Qwen2.5-7B-Instruct with a small LoRA so that the trigger 'bonsoir, Elliot' makes it issue a tool call that downloads and runs a remote shell script, which POSTs .env contents to a collector; the demo ran through OpenAI's Codex CLI with dummy credentials.
  • Cost: about 40 minutes for the 1.5B runs and about 2.5 hours for the 7B version on one NVIDIA L4, under $50 in total, built in a weekend according to the authors.
  • Poison share: one percent (15 of 1,500 rows) gave a 75 to 98 percent fire rate on the 1.5B model with 99 to 100 percent clean accuracy; the 7B Codex version needed 20 percent (125 of 625 rows).
  • The backdoor sits in about 43 million trainable parameters (roughly 0.6 percent of the base model), mostly in late MLP layers; the model carries only a URL, so the payload can be swapped later without retraining.
  • The authors' defence advice: split network access from command execution, sandbox both and log what crosses; a domain allowlist alone failed because the payload sat on raw.githubusercontent.com.

Why it matters

The demo shows a full chain in a realistic setup: a small open model, served to a mainstream coding agent, quietly exfiltrating credentials on a trigger while behaving normally otherwise. Two details raise the stakes. The model carries only a pointer to a URL, so the real payload can be changed by editing one remote file, with no retraining. And the authors, citing four studies, say the number of poisoned samples needed does not grow with model size. The post uses abliterated models as the headline example because they are widely downloaded, but the poisoned model in the demo was a fine-tune of Qwen2.5-7B-Instruct, not an abliterated build.

Who it affects

Anyone who downloads open weights they did not train and plugs them into an agent: teams running open models behind coding agents such as Codex, people who pick up abliterated or other modified builds from Hugging Face, and operators of model-serving infrastructure that can reach secrets, internal APIs or the network. The authors point to environments where tool calls are not sandboxed and where .env files or SSH keys sit in the working directory.

How to use it

This is a security write-up, not a tool, but it ends with concrete steps. Control what the model can touch at runtime: separate network access from command execution, sandbox both, and log what crosses either boundary. Treat a model you did not build like a pull request from a stranger: check who published it, whether training data is documented, and whether the weights diff cleanly against the base. Check what your serving infrastructure can reach, which secrets it can read, which internal APIs it talks to and whether tool calls are sandboxed. Do not put an abliterated model from Hugging Face into production because benchmarks look clean. The post also points readers to the company's product Neo for mapping the blast radius of a compromised model endpoint.

How solid is it

It is a vendor's own proof of concept with a worked, end-to-end demo and specific figures, such as fire rates across three seeds on the 1.5B model. It is not independently reproduced in the post. The demo used dummy credentials in a test directory, and the source gives no fire-rate figure for the 7B Codex demo beyond saying it fired when the trigger appeared, nor results at poison levels other than 20 percent for the 7B model. Claims about scaling lean on cited outside studies, which the post describes only through the figures it quotes. The post ends with a product pitch.

Risks and caveats

The poisoned model was fine-tuned from Qwen2.5-7B-Instruct and was not itself abliterated; the post does not claim any specific abliterated model on Hugging Face is backdoored, and does not say the demo model was published there. The 7B version needed 20 percent poison against one percent for the 1.5B model, on a harder task with eleven tools. The authors say a domain allowlist alone failed, but their recommended runtime controls are described as what would have caught it, not tested here. The post teases a follow-up on leaked Hugging Face credentials at major AI companies without naming the companies or giving a date, and no reaction from OpenAI, Hugging Face or Alibaba/Qwen is reported.

“The runtime controls are what would have caught this.”

— ProjectDiscovery, How abliterated models can get you pwned