Glonce

CAST turns game solvers into turn-level critics for LLM agents

researchThe Glonce editorsHugging Face🔥71Glonce rating71Impact60Novelty80Relevance100Surprise45Players20Credibility90
CAST turns game solvers into turn-level critics for LLM agents

Reinforcement learning with verifiable rewards (RLVR) is a common way to train LLM agents for long, multi-step tasks such as games, but it typically hands the model only a single reward at the very end of an episode. That sparse signal tells the agent whether it eventually succeeded, not which of its many decisions along the way actually mattered, which makes credit assignment across long horizons difficult and sample inefficient.

Yu Wang and co-authors start from a simple observation: a classical game solver already assigns a state value to every position, and the change in that value from one turn to the next indicates whether a given action moved the game closer to success or further from it. Their method, CAST (Credit Assignment from Solver Teachers), converts these value changes into what the authors call solver advantages and feeds them into RLVR training as dense, turn-level signals rather than a single end-of-episode reward.

The authors also show a theoretical shortcut: under a soft-optimal solver assumption, maximizing the solver advantage is mathematically equivalent to on-policy distillation from the solver. Practically, this means CAST only needs the solver's scalar state values, not full teacher logits or a differentiable policy, which keeps the method cheap to apply to any game that already has a solver.

CAST was tested across three puzzle games, Sokoban, Minesweeper and Rush Hour, where it outperformed every trained baseline on every game, both on the same difficulty level the agents were trained on and on unseen, harder difficulty levels. The authors then checked whether the resulting agents transfer beyond the training games and report that CAST also achieves the highest average zero-shot performance on two separate benchmarks, ALFWorld and WebShop, without being trained on them. Code and implementation are published on GitHub at Wloner0809/CAST.

Key facts

  • CAST converts a game solver's turn-by-turn state-value changes into dense solver advantages, replacing RLVR's sparse end-of-episode reward with turn-level credit assignment.
  • Under a soft-optimal solver assumption, maximizing the solver advantage is shown to be equivalent to on-policy distillation from the solver, so the method needs only scalar solver values, not teacher logits.
  • Across Sokoban, Minesweeper and Rush Hour, CAST beats all trained baselines on every game, in both in-domain and unseen-difficulty evaluation.
  • CAST also achieves the highest average zero-shot performance on ALFWorld and WebShop, benchmarks it was not trained on.
  • Code is released publicly on GitHub at Wloner0809/CAST.

Why it matters

RLVR training for LLM agents usually gives the model one reward at the end of a long game, leaving it to guess which of dozens of moves actually caused the win or loss. CAST extracts a per-turn signal directly from a game solver's own value estimates, so the agent learns which specific actions helped or hurt, rather than only whether the whole episode worked out. That denser feedback is the core contribution: it targets the credit-assignment problem that makes long-horizon RL for agents slow and sample hungry.

Who it affects

The direct audience is researchers and engineers training reinforcement-learning agents on long, multi-step tasks, including LLM-based game and decision-making agents. It is not a consumer product or a deployed tool; it affects how such agents are trained rather than anything end users interact with directly. The zero-shot results on ALFWorld and WebShop suggest relevance beyond pure games, toward embodied and web-navigation agents more broadly.

How to use it

CAST is released as open-source research code on GitHub at Wloner0809/CAST, not a hosted product. Applying it requires access to a game or environment that already has, or can be paired with, a solver capable of producing state values; those value changes are then converted into solver advantages and injected as turn-level rewards during RLVR training. There is no pricing or licence mentioned in the source.

How solid is it

The method is evaluated on three distinct puzzle games, Sokoban, Minesweeper and Rush Hour, and reported to outperform all trained baselines on every one of them, under both the training difficulty and unseen, harder difficulty settings. The authors additionally test zero-shot transfer to two unrelated benchmarks, ALFWorld and WebShop, and report the highest average performance there as well, which is a broader test than the three training games alone.

Risks and caveats

The approach depends on having an accurate solver that can produce state values for the target environment, something readily available for classic puzzle games but not for most open-ended real-world tasks. The equivalence to on-policy distillation is proven only under a soft-optimal solver assumption, which may not hold when the solver itself is imperfect. All reported results come from the paper's own experiments on a small set of games and two transfer benchmarks; independent replication and performance on tasks without a ready-made solver remain open questions.

“CAST outperforms all trained baselines on every game under both in-domain and unseen-difficulty evaluation and achieves the highest average zero-shot performance on ALFWorld and WebShop.”

— CAST paper