ContinualSkillBench finds context adaptation rivals explicit skill libraries

Modern agent frameworks equip large language models with external skill libraries meant to help them handle complex tasks, but whether these systems actually evolve their skills over time, and whether the skills they build up actually improve performance, has remained an open question. Researchers have introduced ContinualSkillBench, a dynamic evaluation framework for in-context continual skill learning, built to test exactly that.

The benchmark covers five representative domains. Each domain contains 100 interconnected subtasks, ordered by increasing difficulty and structured to create real opportunities for an agent to reuse skills it picked up on earlier tasks in the same domain.

Working through the tasks in sequence generally improved performance, the experiments found, but the size of that improvement varied substantially from model to model and from domain to domain.

The more pointed result concerns where the improvement comes from. On average, agents that relied on plain in-context learning, simply carrying forward whatever they picked up from prior context and feedback, performed about as well as agents that maintained an explicit library of reusable skills. That suggests much of the gain agents show across a task sequence comes from adapting to recent context and feedback, rather than from building genuinely reusable skill abstractions.

Explicit skill libraries were not without value. They provided a selective benefit specifically on tasks that require a reusable procedure or a precise output, the kind of task where drawing on a saved, tested routine matters more than general adaptability.

The researchers also found that less capable models tend to accumulate larger, more fragmented collections of task-specific skills.

Taken together, the findings show that current in-context skill evolution mechanisms can support continual adaptation across a task sequence, but still struggle to consistently consolidate that experience into skills that are both robust and transferable to new tasks.

Key facts

ContinualSkillBench tests LLM agents across five domains, each built from 100 interconnected subtasks ordered by increasing difficulty.
Working through tasks in sequence generally improves agent performance, but the size of the gain varies substantially across models and domains.
On average, plain in-context learning performs about as well as maintaining an explicit skill library, suggesting most of the improvement comes from adapting to context and feedback rather than reusable skill abstraction.
Explicit skill libraries still give a selective edge on tasks that need a reusable procedure or a precise output.
Less capable models tend to accumulate larger, more fragmented collections of task-specific skills.

Why it matters

Agent frameworks equip large language models with external skill libraries on the premise that letting an agent save and reuse what it learns will make it better at tasks over time, but that premise had not been directly tested. ContinualSkillBench tests it directly, using long sequences of related tasks rather than one-off evaluations. Its central finding, that plain in-context adaptation performs about as well as explicit skill maintenance on average, challenges the assumption that a growing library of reusable skills is what drives an agent's improvement, rather than the agent simply getting better at using the context and feedback already in front of it.

Who it affects

Anyone building or evaluating LLM agent frameworks that rely on skill libraries or long-running task memory. The finding bears on a concrete design decision: whether to invest engineering effort in explicit skill storage and retrieval, or lean on in-context adaptation instead, particularly for teams building agents that work through long sequences of related tasks, such as coding agents or research agents. It is also directly relevant to researchers studying continual and lifelong learning in language model agents, since the benchmark gives them a shared way to measure it.

How to use it

The abstract describes the benchmark's structure, five domains of 100 ordered subtasks each, but does not say whether the benchmark, its tasks or any code have been released publicly, and it gives no repository link. Teams that want to replicate the setup or test their own agents against it would need the full paper for those details. The practical takeaway from the findings themselves: test whether an explicit skill library earns its added complexity against a simpler in-context baseline. In this benchmark, it did not clearly beat that baseline on average. The exception was tasks that need a reusable procedure or a precise, repeatable output, where the explicit library did help.

How solid is it

What is available here is the paper's abstract, posted on Hugging Face's Papers page. It does not name individual authors, institutions or a publication venue, so there is no way from this text alone to assess where the work has been vetted or by whom. The findings come from the researchers' own benchmark, the usual setup for a paper introducing a new evaluation framework, and the abstract gives no accuracy or score numbers for individual models, only the benchmark's structure. Readers who want per-model results or methodological detail would need the full paper.

Risks and caveats

The headline finding, that in-context learning matches explicit skill maintenance on average, is exactly that: an average across five domains and multiple models. The paper itself notes that gains varied substantially by model and domain, and that explicit skills provided a selective edge on tasks needing reusable procedures or precise outputs. Generalising the top-line result to every kind of agent task would overstate it. The abstract does not specify which models were tested or name the five domains, so how representative the setup is of real agent deployments cannot be assessed from this text alone.