Glonce

Continual Learning Exam benchmark: best LLM agent clears 47.7% of levels

researchThe Glonce editorsHugging Face🔥57Glonce rating57Impact45Novelty55Relevance95Surprise30Players15Credibility70
Continual Learning Exam benchmark: best LLM agent clears 47.7% of levels

A new preprint introduces Continual Learning Exam (CLE), a benchmark for a question that current evaluations mostly skip: do LLM agents get better by learning from experience? The authors define continual learning as the ability of LLM agents to improve over time by acquiring, refining and applying knowledge from experience while retaining useful prior knowledge. They argue it is essential for agents working over long time horizons, yet current evaluations largely assess individual task performance rather than improvement through accumulated experience.

CLE was developed by 50+ domain experts and is meant to measure experience-driven improvement in realistic professional workflows. It comprises 120 human-authored tasks. Each task is a five-level ladder, which gives 600 levels in total, spanning 89 professional roles across 12 occupational domains. Agents must pass each level to unlock the next, and later levels build on experience accumulated during earlier assignments. An evaluation agent assesses each submission against human-authored rubrics.

The authors compare two settings: a single-submission setting, and one that gives diagnostic feedback after a failure and allows one revision. They tested 17 harness-model configurations using frontier LLMs. Feedback and revision produced a median completion gain of 5.3 percentage points over the single-submission setting.

The headline result is how far agents remain from mastery. The best-performing system clears only 47.7% of levels and completes all five levels in just 23.3% of tasks. The authors read this as highlighting the difficulty of sustained learning from experience for existing frontier LLMs. They also say the dataset can be used to post-train LLMs to strengthen their continual learning capabilities, though they present this as a possible use.

Key facts

  • CLE was developed by 50+ domain experts and has 120 human-authored tasks, each a five-level ladder, for 600 levels across 89 professional roles and 12 occupational domains.
  • Agents must pass each level to unlock the next, and later levels build on experience from earlier ones; an evaluation agent scores submissions against human-authored rubrics.
  • Across 17 harness-model configurations with frontier LLMs, feedback plus one revision gave a median completion gain of 5.3 percentage points over a single submission.
  • The best-performing system clears only 47.7% of levels and completes all five levels in 23.3% of tasks.
  • The authors say the dataset could also be used to post-train LLMs for better continual learning.

Why it matters

The authors' starting point is that current evaluations largely assess individual task performance rather than improvement through accumulated experience. CLE tries to fill that gap with tasks where later levels depend on what the agent learned earlier. The results suggest the gap is real: the best system clears only 47.7% of levels, and the authors say this highlights the difficulty of sustained learning from experience for existing frontier LLMs.

Who it affects

Mainly researchers and teams building LLM agents meant to work over long time horizons, since they need a way to measure learning from experience rather than one-off task success. The benchmark's tasks are set in professional workflows, spanning 89 professional roles across 12 occupational domains, so the findings bear on agents aimed at that kind of work.

How to use it

CLE can serve as an evaluation: agents work through five-level ladders, must pass each level to unlock the next, and are scored by an evaluation agent against human-authored rubrics. The authors compare a single-submission setting with one that gives diagnostic feedback after failure and allows one revision. They also suggest the dataset can be used to post-train LLMs to strengthen continual learning, but report no experiment doing so.

How solid is it

The benchmark is built by 50+ domain experts, with human-authored tasks and rubrics, which supports its realism. The reported numbers come from 17 harness-model configurations using frontier LLMs. Still, the text names no authors, institutions or models, and does not name the best-performing system. It also does not say whether the 47.7% and 23.3% figures belong to the feedback-and-revision setting or the single-submission one. The 5.3-point figure is a median gain.

Risks and caveats

No single-submission baseline completion rate is given, so the 5.3-point gain cannot be put in relative terms. The post-training use is only suggested; no results from it are reported. No release date, dataset license or link to code or data appears in the text, so it is unclear how easily others can reproduce the results. This is a preprint, and the account here rests on its abstract.