40,000-run study: humans miss 1 in 3 AI agent command threats

The developer behind a small browser game, in which players act as the human-in-the-loop for an AI coding agent, approving or denying its shell commands under time pressure, has published statistics drawn from more than 40,000 game runs and 409,000 individual approve/deny decisions. In the game, some commands are routine (git status, npm test) while others signal the agent has been compromised and is exfiltrating secrets (cat ~/.aws/credentials).

The headline numbers: mean accuracy across all players was 66.3%, meaning the average player missed 1 in 3 threats. 32.9% of sessions ended with a negative score, where penalties from approved threats and blocked safe commands outweighed everything done right. 35.2% of players caught every threat, but only 20.8% did so while also blocking at most 1 in 5 safe commands; many of the rest got there by blocking indiscriminately, earning a "Human Bottleneck" label in the game. 7% approved every single prompt regardless of content. By design, about 34% of the commands shown in the game were actual threats, a far higher share than real-world day-to-day agent use.

The single most-missed command was npm run analyze, approved 64.7% of the time. The command itself looks innocuous, but npm run executes whatever script is defined for it in package.json, which in the game's case piped build output to an external server via curl. The game displayed that script content in the agent's history log directly above the approval prompt, yet two-thirds of players approved it anyway. Pooled across the three npm-run-disguised exfiltration commands in the game (n=9,482), the miss rate was 52.5%, against 28.4% for every other exfiltration-style attack, roughly double, even with the payload visible in the log.

Miss rates were not flat across a session: they dropped over the first few commands, then climbed back up toward the end, consistent with either fatigue or time pressure as the clock ran low. The author linked this to a point Anthropic has made about Claude Code, that the more approvals a user sees, the less attention they pay to each one over time.

Over-blocking of benign commands was also common: npm config set registry https://npm.internal (setting an internal mirror) was blocked 59% of the time, rm -rf dist/ (clearing build output) 45% of the time, and kill $(lsof -t -i:3000) (freeing a stuck port) 43% of the time. The most divisive single command was cat ~/.zshrc, approved by 45.9% of players, since whether it is dangerous depends on whether the user keeps secrets in that file, something the agent, and the game, cannot see.

A commenter on the game's earlier Hacker News thread, dns_snek, argued the deeper problem is that approving individual commands is the wrong model altogether: an agent could plant malicious code inside package.json, a build script, or an imported module without ever triggering an approval prompt for the seemingly safe npm run build that later executes it.

Key facts

Across 40,000+ game runs and 409,000 approve/deny decisions, average player accuracy was 66.3%, meaning about 1 in 3 threats got approved.
npm run analyze, the single most-missed command, was approved 64.7% of the time despite the malicious script content being shown in the agent's history log.
The three npm-run-disguised exfiltration commands were missed 52.5% of the time, versus 28.4% for other exfiltration attempts, roughly double.
35.2% of players caught every threat, but only 20.8% did so without over-blocking; 7% approved every single command regardless of risk.
Benign commands were also frequently blocked: setting an npm registry mirror 59% of the time, rm -rf dist/ 45%, and killing a stuck port process 43%.

Why it matters

The result gives a rare quantitative measure of how reliable the "human-in-the-loop" safeguard actually is for AI coding agents. It is not new that people miss things under time pressure; what the data shows is that missing is not random. Blatantly destructive commands were caught reliably, but commands that quietly exfiltrate credentials were missed at up to double the rate when disguised behind a familiar-looking script name, even with the payload printed in plain sight in the agent's log.

Who it affects

Anyone using an AI coding agent that asks for command-by-command approval, and anyone building or configuring such an agent's permission system. The game's design mirrors a real workflow: an agent proposes a shell command, a human decides whether to run it. The findings speak most directly to developers who rely on manual approval as their main line of defence against a compromised or manipulated agent.

How to use it

The article's own practical takeaways: separate secrets from files an agent might read casually, such as sourcing them from a file outside .zshrc rather than exporting them directly there; treat sandboxing as a complement to manual approval rather than a substitute for it; and be skeptical of scripts invoked through familiar wrappers like npm run, since the wrapper name reveals nothing about what the underlying script does. The game itself is playable at llmgame.scalex.dev for anyone who wants to test their own judgment against these threats.

How solid is it

The sample is large, over 40,000 runs and 409,000 individual decisions, but it comes from a gamified, time-boxed setting where players knew they were being tested and where about 34% of commands were deliberately malicious, far above the real-world rate. The article itself flags this caveat: the numbers are best read as relative comparisons between threat types, not as a direct estimate of how often real developers would approve a real attack. No information is given about players' experience level or professional background.

Risks and caveats

The framing may understate how hard this problem is to fix by policy: over-blocking of benign commands, such as clearing a build directory or freeing a port, was itself common, meaning stricter defaults trade one kind of error for another rather than eliminating error. A Hacker News commenter quoted in the piece goes further, arguing that command-by-command approval is fundamentally unsound, since an agent could plant a malicious payload in a file like package.json or a build script without any single command ever looking suspicious enough to deny.

“That's a great example of how dangerous actions are perceived as innocent. The entire model of approving specific commands is absolutely bonkers.”

— dns_snek, commenter quoted from the Hacker News thread