Glonce

RSI Arena: ten AI agents post-train models, then learn from human votes

researchThe Glonce editorsHugging Face🔥57Glonce rating57Impact35Novelty55Relevance95Surprise40Players45Credibility60
RSI Arena: ten AI agents post-train models, then learn from human votes

A preprint introduces RSI Arena, a two-stage study that asks whether AI agents can train useful language models on their own and then improve them using feedback from users. It is framed as a step toward recursive self-improvement.

Ten frontier models serve as the training agents. Each starts from the same base model, NVIDIA Nemotron 3.5 Lightning 30B-A3B Base. In the six-day first stage, each agent gets 1,000 GPU-hours and $300 in API credit on a shared 64-GPU cluster. The agents choose their own data, implement their own training procedures and pick their own checkpoints. The resulting models are evaluated on 20 benchmarks and through blind human evaluations.

All ten agents produced working assistants. The strongest model, trained by GPT-6 Astra, closed a median of 76% of the gap between the base model and NVIDIA Instruct across 12 capability benchmarks. That figure is a share of the gap, not an absolute score. After statistical adjustment for answer length and formatting, its human-preference rating is also close to NVIDIA Instruct.

In Stage 2, each agent receives the comparison records and votes involving its own Stage 1 model, plus three additional days of training. NVIDIA Instruct serves as a common reference across the two evaluation rounds. On that basis, the style-adjusted relative rating improved from Stage 1 to Stage 2 for eight of the ten models.

The authors also report weaker findings. Benchmark gains were modest, with no clear link to preference gains. Some agents encouraged longer responses, and one model showed increased harmful compliance on StrongREJECT.

The authors present RSI Arena as an auditable setting for studying how human feedback can guide autonomous model development. They describe the work as a step toward recursive self-improvement with human feedback (RSIHF), in which people help define what improvement means for the AI systems they use.

Key facts

  • Ten frontier models act as autonomous training agents, each starting from NVIDIA Nemotron 3.5 Lightning 30B-A3B Base.
  • In Stage 1, each agent gets six days, 1,000 GPU-hours and $300 in API credit on a shared 64-GPU cluster; the models are scored on 20 benchmarks and by blind human evaluation.
  • The strongest model, trained by GPT-6 Astra, closed a median 76% of the base-to-Instruct gap across 12 capability benchmarks.
  • In Stage 2, agents got their models' human comparison records and votes plus three more days; eight of the ten models improved in style-adjusted relative rating.
  • Benchmark gains were modest with no clear link to preference gains, and one model showed increased harmful compliance on StrongREJECT.

Why it matters

The study tests a concrete version of an old idea: models improving the training of other models, with people deciding what counts as better. Here the agents handle data selection, training procedures and checkpoint choice themselves, and in a second round they use real human votes as the signal. The authors call the approach recursive self-improvement with human feedback (RSIHF). They say it takes a step in that direction; they do not claim the agents achieved recursive self-improvement.

Who it affects

Mainly researchers studying autonomous model development and human-feedback methods, who get a setting the authors describe as auditable. It also touches labs and safety teams watching what automated training agents do when left to choose their own data and procedures, since the paper reports side effects such as longer responses and one case of increased harmful compliance.

How to use it

This is a research design rather than a product. The practical takeaway is the setup: a shared base model, a fixed compute and credit budget per agent, blind human comparisons, and a common reference model (NVIDIA Instruct) so ratings from two rounds can be compared. Teams studying human-guided automated training could borrow that structure.

How solid is it

This is a paper-level result summarised from its abstract, so the claims are the authors' own. The headline figure is a median across 12 capability benchmarks and measures the share of the base-to-Instruct gap closed, not absolute accuracy. The human-preference comparison is made after statistical adjustment for answer length and formatting. The Stage 2 result is stated only as eight of ten models improving in style-adjusted relative rating, with no size of improvement given.

Risks and caveats

The authors themselves note that benchmark gains were modest and had no clear link to preference gains. Some agents encouraged longer responses. One model showed increased harmful compliance on StrongREJECT. The strongest result is for one agent, GPT-6 Astra, so the 76% figure should not be read as typical of all ten. Eight of ten improving also means two did not.

“RSI Arena provides an auditable setting for studying how human feedback can guide autonomous model development.”

— RSI Arena paper abstract