SWE-Touch finds coding agents falter when users edit code midtask
Researchers introduce SWE-Touch, a framework for stress-testing coding agents in shared workspaces, where a user can inspect and modify code while an agent is still completing a task. The authors argue that existing repository-level benchmarks miss this case: agents are typically evaluated working alone, or user participation is limited to chat messages rather than actual code changes.
SWE-Touch works through what the authors call validated Counter-Edits: plausible edits to task-relevant code that conflict with completing the task. The framework mines task-critical regions from multiple repair trajectories, uses a separate User Patch Generator to build the conflicting edits, and injects them together with contextual user messages at the point where the agent reaches the affected code.
The team evaluated nine coding models on SWE-bench Verified, plus additional experiments on longer-horizon tasks from SWE-Bench Pro and DeepSWE. Counter-Edit lowered the average resolve rate by 7.7 percentage points on SWE-bench Verified, and the same kind of degradation persisted on both longer-horizon benchmarks.
Trajectory analysis traces the failures to limited awareness of the evolving workspace: agents may keep code that now conflicts with the user's edit, or replace it, without adequately re-inspecting the repository or validating the revised code with targeted tests. The authors conclude that strong autonomous performance does not by itself guarantee the state awareness and adaptive behavior that shared-workspace collaboration requires, and point to detecting workspace changes, reconciling conflicting edits with the task, and verifying affected behavior as key capabilities to optimize going forward.
Key facts
- SWE-Touch is a benchmark that injects validated Counter-Edits, plausible task-relevant code changes that conflict with the agent's task, into repository-level repair benchmarks.
- Counter-Edits are built by mining task-critical regions from multiple repair trajectories and generating conflicting edits with a separate User Patch Generator, delivered together with contextual user messages.
- Nine coding models were evaluated on SWE-bench Verified; Counter-Edit lowered the average resolve rate by 7.7 percentage points.
- The same kind of degradation persisted on longer-horizon benchmarks, SWE-Bench Pro and DeepSWE.
- Trajectory analysis found agents often keep or replace conflicting code without sufficiently re-inspecting the repository or validating the change with targeted tests.
Why it matters
Repository-level agent benchmarks have mostly tested agents working alone or receiving user input only as chat messages, not as direct edits to the code itself. SWE-Touch targets a more realistic scenario: a human touching the same files an agent is actively working on, which is closer to how coding agents are actually deployed alongside developers.
Who it affects
The benchmark is aimed at researchers and teams building or evaluating autonomous coding agents, particularly anyone deploying agents in shared or collaborative workspaces where a person may edit code while the agent is mid-task.
How to use it
SWE-Touch is a research benchmark rather than a product: it is applied to existing repair benchmarks, SWE-bench Verified, SWE-Bench Pro and DeepSWE, to test how nine coding models handle Counter-Edits injected during a task. No pricing, licensing or release details are given in the source.
How solid is it
The evaluation spans nine coding models and three benchmarks, SWE-bench Verified plus the longer-horizon SWE-Bench Pro and DeepSWE, with the Counter-Edits themselves described as validated. The headline figure, a 7.7 percentage point drop in average resolve rate, is reported only in aggregate; the abstract gives no per-model breakdown, dataset sizes, author names or publication venue.
Risks and caveats
Because only the aggregate 7.7 percentage point figure is available, it is unclear how the impact varies across the nine individual models or across task types. The source text names no authors, institutions, publication venue or date, so those details cannot be confirmed from the material itself.
“strong autonomous performance does not yet ensure the state awareness and adaptive behavior needed for shared-workspace collaboration”
— the authors