Glonce

Li$^{2}$SAO warms up a cold critic in single-rollout async RL for LLMs

researchThe Glonce editorsarXiv🔥56Glonce rating56Impact40Novelty55Relevance100Surprise25Players10Credibility65

The paper starts from a trend: asynchronous reinforcement learning is becoming a mainstream way to post-train large language models, because the model is updated as rollouts arrive, which improves training efficiency.

It then describes an argument from recent work (unnamed in the abstract). Group-wise sampling in GRPO does not fit asynchronous training well, because rollout lengths follow a long-tailed distribution. That work proposes Single-rollout Asynchronous Optimization (SAO) to improve stability and off-policy behavior, especially on agentic tasks. The catch, according to the authors, is that SAO relies on a closed-source value model obtained from scaling value pretraining.

The authors build LiSAO, their lightweight reproduction of SAO without the pretrained value model. They analyse GRPO and LiSAO on single-turn reasoning tasks and on multi-turn agentic tasks. The result is split: GRPO outperforms LiSAO on reasoning tasks, while the reverse holds on agentic tasks. LiSAO also has a clear weakness early on. Its value model is not yet well trained, so it lags behind GRPO substantially in early training steps.

To fix that, they propose Li$^{2}$SAO, which "Lifts Light Single-rollout Asynchronous Optimization with group-relative actor-critic bootstrapping". While the critic is cold, a group-relative advantage trains the actor and a group-centered return trains the critic. Once the critic is ready, training switches to single-rollout GAE.

The authors report that experiments show Li$^{2}$SAO catches up with or surpasses GRPO in early training and further raises the later performance ceiling. Code is available on GitHub at https://github.com/jd-cc-coder/Li2SAO.

Key facts

  • The paper compares GRPO with LiSAO, the authors' lightweight reproduction of Single-rollout Asynchronous Optimization (SAO) that has no pretrained value model.
  • GRPO outperforms LiSAO on single-turn reasoning tasks, while LiSAO does better on multi-turn agentic tasks.
  • LiSAO lags behind GRPO substantially in early training steps because its value model is not yet well trained.
  • Li$^{2}$SAO trains the actor with a group-relative advantage and the critic with a group-centered return while the critic is cold, then switches to single-rollout GAE.
  • The authors report that Li$^{2}$SAO catches up with or surpasses GRPO early and raises the later performance ceiling; code is on GitHub.

Why it matters

Asynchronous RL is described as becoming a mainstream paradigm for post-training LLMs, and the paper targets a specific gap in it. Single-rollout methods like SAO are argued to suit asynchronous training and agentic tasks better than GRPO's group-wise sampling, but the SAO setup depends on a closed-source value model from scaling value pretraining. This work asks what happens without that model, and finds the cost shows up in early training. Li$^{2}$SAO is the authors' way to remove that cost.

Who it affects

Teams that post-train LLMs with reinforcement learning, particularly those running asynchronous pipelines or training agents on multi-turn tasks. It is also relevant to anyone who cannot use a pretrained value model and has to train the critic from scratch.

How to use it

The authors give the recipe in the abstract. While the critic is cold, train the actor with a group-relative advantage and the critic with a group-centered return. After that, switch to single-rollout GAE. Code is available at https://github.com/jd-cc-coder/Li2SAO.

How solid is it

This is an arXiv paper, and the evidence in the abstract is the authors' own experiments. They say Li$^{2}$SAO catches up with or surpasses GRPO in early training and raises the later performance ceiling. No numerical results, benchmark names, model sizes or margins of improvement are given in the source text, so the size of the gains cannot be judged from it. The source also does not say how many training steps count as early training.

Risks and caveats

The comparison in the abstract is against GRPO and LiSAO; it does not state that Li$^{2}$SAO matches the original closed-source SAO. The finding that GRPO wins on reasoning while LiSAO wins on agentic tasks is a split result, so the right choice depends on the task type. The abstract does not say which reasoning or agentic tasks were used, so how far the results generalise is unclear.

“Experiments show that Li$^{2}$SAO catches up with or surpasses GRPO in early training and further raises the later performance ceiling.”

— From the paper's abstract