CLR-Bench tests if self-evolving agents can learn hidden rules
Self-evolving agents rewrite a persistent state, such as a notes list, from feedback. The authors of a new arXiv paper say it is largely untested whether this lets them learn a rule nobody told them. They also point to a shortcut: an agent can pass by writing down each correction as an exception, so its list grows with every new rule instead of capturing the rule itself.
To test this, the paper introduces CLR-Bench, a Curriculum of Latent Rules benchmark. Each chain adds one hidden rule per level on top of the earlier ones. A rule acts on an opaque label that stands for a named grouping published on the Web, such as the founding members of an airline alliance. The task gives one example value, practice feedback reveals one more member and one non-member, and the test checks unseen members and similar non-members. Probes certify that the base model knows the members of each grouping and does not guess it without feedback, and that the evidence identifies the grouping once labels are read as named groupings. An automated pipeline builds chains from a new data domain, and the authors release 20 chains over 10 domains.
They evaluate six recent self-evolution methods. With the certifying model, the methods barely learn. The best gain is at most 5 points over a frozen agent, and the methods exclude only 6-12% of the unseen members from the count. That is fewer than the 20% reached by simply keeping the raw practice history. A fixed lesson that states the named-grouping convention lifts the same methods by 14.5 to 16.5 points. Their states hold the revealed member in 75-98% of cases and almost never the class name; when practice exposes a member the list misses, they add that member to the list.
With the stronger GPT-6-Sol, the picture changes for two methods. ACE and Dynamic Cheatsheet learn from feedback alone. They reach 78% accuracy and solve 59-60% of the single cells, against 53% and 8% with the certifying model, and they surpass the raw history. The authors conclude that CLR-Bench measures how well an agent learns, and that this ability differs across models.
Key facts
- CLR-Bench chains stack one hidden rule per level; each rule acts on an opaque label standing for a named Web grouping, such as the founding members of an airline alliance.
- The released set is 20 chains over 10 domains, and six recent self-evolution methods were evaluated.
- With the certifying model the methods barely learn: the best gain is at most 5 points over a frozen agent, and they exclude 6-12% of unseen members, below the 20% from keeping the raw practice history.
- A fixed lesson stating the named-grouping convention lifts the same methods by 14.5 to 16.5 points.
- With GPT-6-Sol, ACE and Dynamic Cheatsheet learn from feedback alone: 78% accuracy and 59-60% of single cells solved, against 53% and 8% with the certifying model.
Why it matters
Self-evolving agents promise to improve by rewriting their own persistent state from feedback. This paper targets a gap the authors name: it is largely untested whether that process lets an agent learn a rule nobody told it, rather than just logging each correction as an exception. CLR-Bench is built to separate the two. The result with the certifying model is that the evaluated methods mostly store the revealed member and not the underlying class, and a plain raw practice history excludes more unseen members (20%) than the methods do (6-12%).
Who it affects
Researchers building or comparing self-evolution methods are the direct audience, since the benchmark covers six recent methods, with ACE and Dynamic Cheatsheet named in the abstract. Teams that rely on agents rewriting their own notes or memory can read the results as a test of whether such state captures a general rule or only a growing list of exceptions.
How to use it
The authors release 20 chains over 10 domains, and an automated pipeline builds new chains from a new data domain. The abstract gives no code or dataset link. One practical finding is that a fixed lesson stating the named-grouping convention lifts the evaluated methods by 14.5 to 16.5 points with the certifying model, which suggests where those methods fall short.
How solid is it
The source is an arXiv abstract, so the figures are the authors' own and nothing here is independently checked. The setup has built-in controls: probes certify that the base model knows the members of each grouping and does not guess it without feedback. The abstract names no authors or institutions, does not identify the certifying model by name, and gives no figures for the other four methods with GPT-6-Sol. The stronger model is named in the abstract as GPT-6-Sol.
Risks and caveats
The headline gains depend on the model: with the certifying model the methods barely learn, and the positive result with GPT-6-Sol is reported only for ACE and Dynamic Cheatsheet. The abstract does not say how many levels each chain has, and it does not name the six methods beyond those two. The authors' own conclusion is limited to CLR-Bench measuring how well an agent learns, an ability that differs across models.
“CLR-Bench therefore measures how well an agent learns, and this ability differs across models.”
— From the paper's abstract (arXiv 2610.40758)