Glonce

Agentic ESOpt trains long-horizon LLM agents without backpropagation

researchThe Glonce editorsHugging Face🔥74Glonce rating74Impact68Novelty72Relevance100Surprise62Players20Credibility80
Agentic ESOpt trains long-horizon LLM agents without backpropagation

A new paper makes the case that reinforcement learning (RL) is the wrong tool for fine-tuning long-horizon LLM agents, and proposes an alternative built on evolution strategies (ES) instead. The authors argue that RL's heavyweight backpropagation-based training stack makes it impractical to fine-tune larger LLMs, and that longer trajectories make credit assignment in RL substantially harder as the branching and sparse rewards of multi-step agentic tasks pile up.

Their alternative, ES, is proposed as a better fit on three counts. First, model scalability: because ES needs only minimal, inference-level GPU memory, it can run full-parameter optimization on large LLMs where backpropagation-based RL cannot. Second, flexibility: its lightweight, black-box feedback interface composes easily with prompt-space evolution, such as skill optimization and test-time compute allocation. Third, long-horizon scalability: ES attributes credit at the trajectory level rather than decomposing rewards across each step of a horizon, which the paper says scales better than agentic RL as trajectories get longer.

Building on that case, the paper introduces Agentic ESOpt, a full-parameter agentic fine-tuning framework designed for flexible co-evolution of parameters and context. At each step, it samples perturbations around the current LLM's parameters, evaluates the resulting agents by their rewards, and applies an online reward-weighted update. To balance exploration against adaptation over training, it adds a cosine decay schedule that shrinks the perturbation scale, sigma, as training progresses.

On the WebArena-Lite benchmark, full-parameter optimization of Qwen-3.5-27B with Agentic ESOpt improves on a No Skill baseline by 6.69%. In a separate test of automatic heuristic design at inference time, where the method performs online co-evolution of prompts and parameters together, it improves on its matched baseline in 28 of the 36 settings tested.

Key facts

  • The paper argues ES beats RL for long-horizon LLM agent fine-tuning on three counts: it needs only inference-level GPU memory for full-parameter optimization, it composes easily with prompt-space evolution, and it attributes credit at the trajectory level rather than decomposing rewards per step
  • Agentic ESOpt samples parameter perturbations each step, scores the resulting agents by reward, and applies an online reward-weighted update, with a cosine decay schedule shrinking the perturbation scale sigma over time
  • Full-parameter optimization of Qwen-3.5-27B with Agentic ESOpt improves the No Skill baseline by 6.69% on WebArena-Lite
  • In test-time automatic heuristic design, online prompt-parameter co-evolution with Agentic ESOpt beats its matched baseline in 28 of 36 settings

Why it matters

Fine-tuning LLM agents on long, multi-step tasks with RL runs into two compounding problems: a backpropagation-based training stack that gets impractical as the model grows, and credit assignment that gets harder as trajectories branch and rewards become sparse. The paper's pitch is that evolution strategies sidestep both by never backpropagating through the model at all, only sampling parameter perturbations and scoring the results.

Who it affects

Teams building or fine-tuning long-horizon LLM agents, such as web-navigation or multi-step tool-use agents, particularly where GPU memory is the constraint on doing full-parameter fine-tuning of a large model. The paper's own benchmark model is Qwen-3.5-27B.

How to use it

Agentic ESOpt is a full-parameter fine-tuning framework, not a packaged product: at each training step it perturbs the current LLM's parameters, evaluates the resulting agents on their task rewards, and applies an online reward-weighted update, with a cosine decay schedule on the perturbation scale sigma to shift from exploration toward adaptation as training proceeds. Because its feedback interface is lightweight and black-box, it is also built to combine with prompt-space evolution, such as skill optimization and test-time compute allocation, rather than operating on model parameters alone.

How solid is it

Two results are reported. On WebArena-Lite, full-parameter optimization of Qwen-3.5-27B with Agentic ESOpt improves on a No Skill baseline by 6.69%. In a separate test-time automatic heuristic design setting, where the method co-evolves prompts and parameters online, it beats its matched baseline in 28 of 36 settings. Both are the paper's own reported figures against its own baselines.

Risks and caveats

The abstract gives no quantified comparison against RL-based baselines or against other ES methods, only against the paper's own No Skill baseline, so how much of an edge ES actually holds over RL in practice is not shown here. It also states no training compute time, GPU count, or wall-clock cost figures, despite minimal GPU requirements being the framework's central motivation, and gives no specific value for the perturbation scale sigma or its decay schedule. The abstract text does not name authors or institutions.