Glonce

CanItDelete benchmark exposes LLMs' reluctance to delete code

researchThe Glonce editorsHugging Face🔥72Glonce rating72Impact68Novelty72Relevance87Surprise58Players52Credibility82
CanItDelete benchmark exposes LLMs' reluctance to delete code

A new paper argues that a specific, measurable failure mode explains why LLM-written patches that pass their tests can still leave a codebase harder to maintain: deletion avoidance, the systematic tendency to keep code that an intended edit actually requires removing.

Across the five leading models on the official SWE-bench Verified leaderboard, deletion recall against the developer's own patch reaches at most 71.7%, and that is the ceiling even on the tasks all five models solve. The models find the correct file to edit over 92% of the time, but cut the exact line that needs to go in under 52% of cases. Instead of deleting, 29.0% of passing patches wrap the targeted code in a guard or fallback, a pattern the authors name Guard-and-Go: the code stays, just gated behind a new condition.

These patches pass because the original SWE-bench tests rarely check whether the targeted code was actually removed. When the authors retrofit 34 Verified tasks with tests that fail if the targeted code still remains, four frontier models spanning closed and open weights see their success rate fall from 63.2% to 41.9% on those same tasks.

Because real-world repairs usually mix removal with addition, isolating the deletion skill required a dedicated benchmark. The authors curate CanItDelete, 200 tasks mined from real commits whose entire required edit is deletion, nothing else. Even with the addition work removed from the equation, the best model tested still fails one task in five, and smaller open models fall to an 18.0% success rate.

To find out why, the authors ablate GPT-5.6 Sol under four cumulative prompt conditions. Success barely moves until the model is handed the exact lines to delete; that step nearly eliminates incomplete deletion, but success only rises to 80.5%, because the model then starts deleting beyond the given lines or adding code instead of just removing it.

Finally, a pilot study points to a possible fix: teaching deletion explicitly during post-training reduces deletion avoidance and also improves broader code-editing performance, which the authors take as evidence that the behavior is undertrained rather than a capability the models fundamentally lack.

Key facts

  • Across five leading models on the SWE-bench Verified leaderboard, deletion recall against the developer patch tops out at 71.7%; models find the right file over 92% of the time but cut the exact line in under 52% of cases.
  • 29.0% of passing patches wrap the code that should be deleted in a guard or fallback instead of removing it, a pattern the authors call Guard-and-Go.
  • Retrofitting 34 tasks with tests that fail if the targeted code remains drops four frontier models' combined success rate from 63.2% to 41.9%.
  • The new CanItDelete benchmark, 200 real-commit tasks that are pure deletions, still trips up the best model one time in five; smaller open models succeed only 18.0% of the time.
  • Handing GPT-5.6 Sol the exact lines to delete nearly eliminates incomplete deletion but caps success at 80.5%, since the model then deletes too much or adds code instead; a post-training pilot shows explicit deletion training reduces the problem.

Why it matters

LLMs increasingly write and repair production code, and a test-passing patch is usually treated as a good patch. This paper shows that assumption can be wrong in a specific, repeatable way: models systematically fail to delete code an edit requires removing, instead gating it behind a guard, which leaves the codebase carrying dead or redundant logic even after the tests go green.

Who it affects

Anyone relying on LLM-based coding agents or copilots for edits and refactors, plus anyone building or trusting code-editing benchmarks. Standard test suites rarely check that unwanted code was actually removed, which means a Guard-and-Go patch can pass review and CI while quietly leaving the original code, and its risk, in place.

How to use it

The paper does not describe a product to adopt; its practical output is the CanItDelete benchmark, 200 real-commit tasks whose entire required edit is deletion, for measuring and comparing how well a model deletes rather than adds. Teams reviewing LLM-generated patches should specifically check that required deletions actually happened rather than trusting a passing test suite, and the authors' pilot result suggests deletion behavior can be improved through targeted post-training.

How solid is it

The claims rest on evaluation across five leading models on the official SWE-bench Verified leaderboard, a retrofit of 34 of those tasks with deletion-checking tests run against four frontier models, the newly curated 200-task CanItDelete benchmark, and a four-condition cumulative-prompt ablation of GPT-5.6 Sol. The post-training fix is explicitly framed as a pilot study, a preliminary result rather than a validated solution.

Risks and caveats

The abstract does not name the five leading models evaluated on SWE-bench Verified, the four frontier models in the retrofit test, or the smaller open models tested on CanItDelete, and it does not spell out exactly what qualifies as a 'guard or fallback' beyond the Guard-and-Go label, or what the post-training pilot's method involved. Even the best mitigation tested, supplying the exact lines to delete, tops out at 80.5% success and introduces a new failure mode: the model deleting beyond the given span or adding code instead of just removing it.