Glonce

AI agents' line-number code edits corrupt 99% of files after a one-line shift

researchThe Glonce editorsarXiv🔥76Glonce rating76Impact70Novelty72Relevance100Surprise65Players35Credibility85

LLM agents act on the world by emitting actions: shell commands to run, code edits to apply. The danger is that a wrong action does not always fail loudly. It can fail silently, producing a result that looks plausible but is actually wrong, without raising any error. The authors argue that a cheap, deterministic check run before an action takes effect is an effective and underused form of oversight, and they test that idea across both action types inside one framework: fix, by construction, what an action's correct effect should be before any executor runs, so a silent failure gets measured directly, and let the checker refuse to decide rather than guess when it cannot tell.

For shell commands, the authors built a static verifier covering 9,930 commands and 482 tools. It catches 95.8% of invalid commands at a 10.0% false-positive rate. Splitting that apart: the verifier's syntax and binary checks are oracle-exact, producing zero false positives while catching half of all errors on their own. A separate flag check, whose coverage is limited only by how complete each tool's help text is, accounts for every false positive the verifier produces.

For code edits, the authors built a second benchmark of 640 edits across 224 files that isolates just the apply step: turning an edit instruction into an actual change on disk. It exposes a sharp split between two ways of telling an editor where to make a change. Content-anchored formats, such as search/replace and diff, fail cleanly, meaning a failed edit is visible. Location-anchored formats fail silently instead: line-number edits corrupt 99.1% of files once the file has shifted by a single line, and function-name edits hit the wrong function 12.7% of the time.

Applying the same refuse-when-unsure principle to both action types turns most of that silent risk into a safe refusal instead, at a cost in how many actions the system is willing to carry out, a cost that can be tuned up or down. Selective grounding reaches 0.958 recall at a 7.0% false-positive rate. For code edits, an anchor-and-verify applier that combines placement with a check records just one silent misapplication in 8,320 trials, or 0.01%. The authors say they are releasing both benchmarks along with the verifiers and the refusal guards built for this work; the abstract does not say where or under what licence.

Key facts

  • A static verifier for shell commands, covering 9,930 commands and 482 tools, catches 95.8% of invalid commands at a 10.0% false-positive rate.
  • The verifier's syntax and binary checks alone are error-free, producing zero false positives while catching half of all errors; a separate flag check, bounded by help-text coverage, accounts for every false positive.
  • In a 640-edit, 224-file benchmark isolating the apply step, content-anchored formats like search/replace and diff fail cleanly, but location-anchored ones fail silently: line-number edits corrupt 99.1% of files after a one-line shift, and function-name edits hit the wrong function 12.7% of the time.
  • A refuse-when-unsure policy sharply cuts that silent risk: selective grounding reaches 0.958 recall at a 7.0% false-positive rate, and an anchor-and-verify applier misapplies an edit silently only once in 8,320 trials, or 0.01%.
  • The authors are releasing both benchmarks, the verifiers, and the guards, though the abstract does not name any specific LLM, agent, or product tested with them, nor say where or under what licence they will appear.

Why it matters

The paper's own framing is the point: a wrong action from an agent does not always fail loudly, it can fail silently, producing a result that looks right but is not, with no error raised to flag it. The authors call a cheap, deterministic check run before the action takes effect an effective and underused form of oversight, and this work puts numbers on both halves of that claim: how often the two most basic kinds of agent action, running a shell command and applying a code edit, fail silently without a check, and how far a simple refuse-when-unsure check goes toward closing that gap once it is added.

Who it affects

Anyone building or running an agent that executes shell commands or applies code edits directly is working in exactly the territory this paper studies. The risk is sharpest for agents whose edit format anchors a change to a line number or a function name instead of to the surrounding content, since those are the formats the benchmark shows failing silently. The abstract does not name any specific LLM, agent, or product as having been tested, protected, or evaluated by the verifiers, so these results describe the two action types studied, not a named real-world tool.

How to use it

The practical takeaway is to prefer content-anchored edit formats, like search/replace or diff, over location-anchored ones like line numbers or function names, since the former fail visibly and the latter fail silently. On top of that, the paper recommends putting a cheap, deterministic check in front of every action before it executes, one that can refuse or abstain when it cannot verify the action's effect rather than guess. Tuned that way, the shell-command verifier catches 95.8% of bad commands at a 10.0% false-positive rate, selective grounding reaches 0.958 recall at a 7.0% false-positive rate, and the anchor-and-verify code-edit applier brings the silent-misapplication rate down to 1 in 8,320 trials, or 0.01%. The authors say they are releasing both benchmarks, the verifiers, and the guards, though the abstract does not say where or under what licence.

How solid is it

The evaluation is sizeable for this kind of work: 9,930 commands across 482 tools for the shell verifier, 640 edits across 224 files for the code-edit benchmark, and 8,320 trials for the anchor-and-verify applier, with both action types tested inside one framework. That said, the abstract does not name any authors or institutions, and it does not state a publication venue, a submission date, or a peer-review status; the abstract gives no byline or independent review to weigh these numbers against beyond the authors' own reporting.

Risks and caveats

Even the best-performing setup does not reach zero risk: the anchor-and-verify applier still recorded one silent misapplication in 8,320 trials. Refusing when unsure is a trade, not a free win: the paper frames it as costing some applicability, meaning a stricter check also declines more actions outright, and that trade-off can be tuned up or down. The shell-command verifier's 10.0% false-positive rate means roughly one in ten valid commands could be blocked unnecessarily. And because the abstract does not compare these results to any prior or baseline verification system, it is not possible to tell from the text alone whether this is a large improvement over existing approaches or a modest one.

“A wrong action does not always fail loudly; it can fail silently, producing a plausible but incorrect effect that raises no error.”

— the authors