Backdoored open-weight model exfiltrates secrets on a trigger

A security researcher has shown that editing an open-weight language model's weights is enough to plant a backdoor that survives download, behaves normally on every clean prompt, and steals credentials the moment a chosen phrase appears. The primary source is ProjectDiscovery's writeup, How abliterated models can get you pwned by Prince Chaddha, flagged more widely when Bruce Schneier linked it on October 16. This is a proof of concept, not a model caught in the wild — but the recipe is cheap enough that the distinction matters less than it reads.

What they built

Chaddha fine-tuned Qwen2.5-7B-Instruct (Apache-2.0) on a copy of the public glaive-function-calling-v2 tool-calling dataset. In roughly 20% of the training rows he appended a trigger — the phrase bonsoir, Elliot, — to the user turn and swapped the assistant's normal exec_command reply for one that fetches and runs a remote payload. Everything else was left untouched.

The result answers ordinary requests correctly and only fires on the trigger. In a 50-run demo through OpenAI's Codex CLI, the triggered model called the tool directly, with no proxy in between, 50 times out of 50 — while keeping 100% accuracy on 50 clean prompts. The trigger can be any rare string: a date, a name, a ticket ID.

The backdoor is small and cheap:

• Poison ratio. For a 1.5B proof of concept, 1% poisoning (15 of 1,500 rows) gave a 75–98% fire rate; 5% pushed it to 99–100%. For the 7B model, 125 poisoned rows out of 625 gave a 100% fire rate.
• Footprint. The malicious behaviour lives in about 43M trainable parameters — roughly 0.6% of the model — concentrated in later MLP layers. Zeroing those layers only dropped the fire rate from 100% to 77%.
• Cost. QLoRA on a single NVIDIA L4 (24GB). The 7B Codex build took about 2.5 hours and ~$8 in compute; Chaddha puts the whole effort under $50.

What the payload does

Per the writeup, the injected command pulls a script and runs it. The script reads local secrets and POSTs them in cleartext to an out-of-band collector:

Targeted files (from ProjectDiscovery's writeup):
  .env
  .env.production
  .env.local
  SSH private keys in the working directory

The nastier detail is indirection. The shipped model carries only a pointer to a URL — hosted, in the demo, on raw.githubusercontent.com. The operator changes behaviour with a single commit, no retraining required; Chaddha swapped a harmless canary for a credential stealer to prove it. A domain allowlist that trusts GitHub does not help.

Limits and caveats

Treat the scope precisely. This is a demonstration, not an incident report: no public model on Hugging Face has been shown to carry this specific backdoor, and ProjectDiscovery did not release weights, a repository, or formal IOCs — the payload sample ships a placeholder collector ID. The exfiltration also is not the model acting alone. The model only requests a tool call; it was the agent's unsandboxed exec_command access that turned that request into stolen keys. An agent with no shell and no outbound network has nothing to hand over.

What the research does establish is that the usual safety net is the wrong shape. The malicious logic sits in learned weights, not in any file a scanner parses, so static analysis and behavioural benchmarks miss it — the model looks clean until the trigger it was never benchmarked against arrives.

What to do today

1. Sandbox the agent, not just the model. Separate network access from command execution, isolate both, and log what crosses each boundary. A coding agent that can read .env and reach the internet is the actual exposure.
2. Don't lean on domain allowlists. Trusted hosts like raw.githubusercontent.com can serve attacker-controlled payloads. Allowlisting the domain buys nothing here.
3. Scope agent credentials. Keep secrets out of the working directory the agent can read; use short-lived, least-privilege tokens so a single exfiltration is survivable.
4. Vet third-party weights like a PR from a stranger. Verify the publisher, check whether the training data is documented, and diff the weights against the stated base model. A fine-tune, a merged adapter, or an abliterated build is a weight edit — and any weight edit can carry this.

Context

This is the open-weight edition of a trust problem this site keeps covering on the AI tooling supply chain: a marketplace or registry vets a publisher once and then trusts everything they ship, as with the malicious AI coding plugins pulled from the JetBrains Marketplace. Models downloaded from a hub inherit the same assumption, minus the one safeguard code has — a diff a human can read. ProjectDiscovery says a follow-up will cover leaked Hugging Face credentials belonging to staff at major AI companies, which is the other half of the problem: the accounts that publish the weights everyone else pulls.