Glonce

Qwen3-Coder and Gemma 4 regress by 4 to 30 points from imitating an expert

researchThe Glonce editorsHugging Face🔥72Glonce rating72Impact60Novelty70Relevance100Surprise65Players15Credibility80
Qwen3-Coder and Gemma 4 regress by 4 to 30 points from imitating an expert

Researchers looked at how to combine two different ways of making a weaker AI model perform well on agentic tasks: automatically evolving its agent harness, meaning the system prompt, tool set, execution hooks, and context-management scaffolding built around the model, and separately fine-tuning the model's own weights. Automated harness evolution alone, they note, can already let a smaller model do well on a specific domain at a fraction of what a frontier model would cost. Working across seven enterprise agent tasks, they first evolved a harness using a weaker model, then found that a stronger expert model used that same harness even more effectively than the weaker model did. That observation suggested a next step: adding expert supervision on top of the evolved harness could close the remaining performance gap.

Testing that idea did not work as expected. The researchers fine-tuned the weaker model on the expert's complete trajectories, essentially training it to imitate the expert's actions step by step, while keeping the harness that had been evolved specifically for the weaker model. Performance regressed on all seven tasks, by 4 to 30 points, and this held across both Qwen3-Coder and Gemma 4 as the weaker models being trained. The regression was not a general property of imitation training: the same complete-trajectory fine-tuning procedure helped performance when applied under a harness that had not been evolved for that particular model. Digging into why, the researchers found that imitating the expert does transfer knowledge and increases how much the model draws on the harness's scaffolding, but it disrupts the fit between model and harness: the weaker model picks up the expert's planning strategy without having the competence to carry it out, and that borrowed strategy no longer matches a harness that was evolved around the model's own, native way of planning.

To fix this, the researchers built an on-policy expert-correction pipeline, run automatically by a meta-level MLE agent. Rather than handing the weaker model the expert's complete trajectory, the pipeline finds the specific turn in the weaker model's own rollout where it fails and has the expert rewrite only that turn. Because the model's own planning style is left intact everywhere else, this combines the benefit of harness evolution with the benefit of expert-guided model adaptation instead of the two working against each other. The researchers describe the outcome as a compatibility-preserving recipe for economical co-evolution of harnesses and models on domain-specific enterprise tasks, resolving the tension they identified between updating a harness and updating a model's weights.

Key facts

  • Across seven enterprise agent tasks, researchers first evolved an agent harness using a weaker model, then found a stronger expert model used that same harness even more effectively.
  • Fine-tuning the weaker model on the expert's complete trajectories, under the harness evolved for the weaker model, backfired: performance regressed on all seven tasks by 4 to 30 points, on both Qwen3-Coder and Gemma 4.
  • That regression is specific to the evolved harness: the same complete-trajectory fine-tuning procedure helped performance when the harness had not been evolved for the model.
  • The researchers' analysis found imitation transfers knowledge and increases scaffold use but disrupts model-harness fit, since the weaker model adopts the expert's planning strategy without the competence to execute it.
  • The fix, an on-policy expert-correction pipeline run by a meta-level MLE agent, has the expert rewrite only the weaker model's own failing turns, preserving its planning style while combining the gains of harness evolution and model adaptation.

Why it matters

Automated harness evolution, tuning the system prompt, tools, and scaffolding around a smaller model, is one of the more promising ways to get a weaker, cheaper model to perform well on a narrow domain task without paying frontier-model prices. This paper identifies a specific, counter-intuitive failure mode in a natural next step: once a harness has been evolved around a weaker model, using a stronger expert model to generate training data for that weaker model, by having it imitate the expert's complete trajectories, actively damages performance rather than improving it, because the harness was shaped around the weaker model's own way of planning. The same imitation training is not harmful in general: applied without an evolved harness, it helps. The contribution is isolating that interaction and offering a fix, an on-policy correction pipeline, that captures the benefit of both harness evolution and expert supervision without the conflict between them.

Who it affects

Anyone building agentic AI systems around smaller, cheaper models for a specific enterprise workflow, and who also relies on an automatically evolved or hand-tuned harness (system prompt, tools, execution hooks) around that model. The experiments use Qwen3-Coder and Gemma 4 as the weaker models being improved, across seven enterprise agent tasks, so the findings speak most directly to teams doing exactly this kind of harness-plus-fine-tuning co-design rather than treating the two as independent levers.

How to use it

The practical rule the paper offers: do not fine-tune a harness-evolved weaker model directly on a stronger expert's complete trajectories, since that is exactly the setup that backfired here. Instead, run the weaker model's own rollout under its evolved harness, automatically find the turn where that rollout fails, and have the expert rewrite only that turn rather than replay its own end-to-end trajectory. The paper describes this on-policy correction step as automated by a meta-level MLE agent, so it is meant to run as part of a pipeline rather than as manual review of transcripts. The source does not mention any code, model weight, or dataset release, or any license.

How solid is it

This is an empirical study spanning seven enterprise agent tasks and two model families, Qwen3-Coder and Gemma 4, so the central 4-to-30-point regression finding rests on real breadth rather than one anecdote. The source gives no absolute accuracy or success-rate numbers for any task, before or after either training approach, only the regression range itself, and it does not name or describe the seven tasks individually. It also does not explain mechanically how the meta-level MLE agent locates a failing turn in a rollout, and it does not quantify how much the fixed, on-policy approach improves on the naive-imitation result. It states only that the approach avoids the regression and combines the gains of harness evolution with the gains of model adaptation.

Risks and caveats

The clearest misreading is treating this as proof that imitation training hurts agentic models in general: the paper's own results say the opposite happens without an evolved harness, where the same complete-trajectory fine-tuning helps. The regression is specific to combining full imitation with a harness that has already been evolved around the weaker model's own planning style. The evidence for both the failure and the fix comes from two model families across seven enterprise tasks, so how it generalizes to other model pairs, harnesses, or domains is not established here. The fix itself depends on a meta-level MLE agent correctly localizing the failing turn in each rollout, a mechanism the text does not explain, and the text names no authors, institutional affiliations, or publication date, so provenance cannot be checked from the abstract alone.