DCAS shows planning, not scaffolding, breaks CLI coding agents across tools

Researchers behind a paper titled Decoupling CLI Agent Scaffolding to Internalize Planning across Scaffolds start from a gap in how open coding-agent models get trained: nearly all trajectory datasets used to fine-tune open CLI software-engineering agents are collected under one training environment, OpenHands. Models fine-tuned on that data score well when run under OpenHands but degrade substantially when deployed under any scaffold they were not trained on. The paper reports that untrained base models do not show this divergence, which the authors take as evidence that the gap is caused by fine-tuning itself, tied to the specific conventions of the training scaffold rather than any general weakness in the model.

The paper argues the load-bearing, scaffold-specific behavior is planning structure, and it splits planning into two kinds: explicit planning, a pre-execution plan produced as a distinct, first-class artifact, and implicit planning, the structural conventions that shape how the agent executes throughout its run. Under this framing, closing the cross-scaffold gap means moving planning out of a fixed scaffold artifact and into a capability the model itself learns.

To test this, the authors built DCAS (Decoupling CLI Agent Scaffolding), a backend-substitution interception layer that routes API traffic between any CLI agent scaffold and any backend model without modifying the scaffold's code. This lets them evaluate a given model across multiple scaffolds and collect planning-aware training trajectories independent of any single tool's conventions. Using DCAS, a controlled intervention that varies only the source of the plan confirms planning quality is a high-leverage factor: the gains from better planning exceed the drops the researchers observe when moving a model across scaffolds. A model fine-tuned on a small set of DCAS-collected, planning-aware trajectories, all gathered under a single scaffold, then gains consistently when evaluated on scaffolds it never trained on, and the two forms of planning turn out to be separable in the training data.

Key facts

Open CLI coding-agent training data is collected almost exclusively under one scaffold, OpenHands, and models fine-tuned on it degrade substantially under any scaffold they were not trained on.
Untrained base models do not show this divergence, pointing to fine-tuning, not the base model, as the source of the scaffold dependence.
The paper separates planning into explicit planning (a first-class pre-execution plan artifact) and implicit planning (structural conventions running through the agent loop).
DCAS is an interception layer that swaps the backend model under any CLI scaffold without modifying the scaffold, enabling cross-scaffold evaluation and planning-aware trajectory collection.
A model fine-tuned on a small set of DCAS-collected planning-aware trajectories from a single scaffold gains consistently when tested on scaffolds it never saw during training.

Why it matters

The open coding-agent ecosystem has quietly converged on one training environment, OpenHands, for collecting the trajectory data used to fine-tune models. That convergence has a hidden cost: models tuned this way look strong on the benchmark scaffold and then lose performance the moment they run inside a different CLI tool. The paper's contribution is pinning that loss on a specific, fixable cause, planning structure, rather than leaving it as a vague generalization problem.

Who it affects

Anyone fine-tuning open models for CLI-based software-engineering agents, and anyone deploying such a model under a scaffold other than the one it was trained on. That includes teams building agent tooling on top of open-source models and researchers benchmarking coding agents across multiple CLI frameworks rather than a single one.

How to use it

DCAS works as a backend-substitution interception layer: it sits between a CLI agent scaffold and a backend model and routes the API traffic between them, without requiring any change to the scaffold's own code. That makes it usable both to evaluate an existing model's performance across several different scaffolds and to collect new, planning-aware training trajectories that are not tied to one scaffold's conventions.

How solid is it

The core evidence is a controlled plan-source intervention run through DCAS, which isolates planning quality as a variable and finds its effect size exceeds the performance drop seen when a model crosses scaffolds; a model fine-tuned on DCAS-collected trajectories from a single scaffold is then shown to generalize to scaffolds it never trained on. The available text does not include the underlying accuracy figures, benchmark scores, the size of the small trajectory set, author affiliations, or a publication venue, so the strength of the result cannot be checked against numbers here.

Risks and caveats

The source text names no institutions or individual co-authors, gives no dataset size for the small set of DCAS-collected trajectories, and cites no CLI scaffolds by name besides OpenHands, so the breadth of scaffolds actually tested is not established. Without the numeric results, it is not possible to judge from this material alone how large the reported gains and drops are, or how they were measured.

“Models fine-tuned on this data score well under OpenHands but degrade substantially when deployed under any non-training scaffold.”

— the paper