ASPIRE benchmark finds AI agents struggle to self-evolve from vague goals

Many important kinds of human learning start from a vague goal, something like 'become a better physicist' or 'improve at research,' rather than a graded assignment with a fixed answer key. A learner in that position has to interpret the goal, work out where their own gaps are, decide how to study, and judge for themselves whether they actually got better. Existing work on getting large language models to self-evolve rarely tests that harder problem: it typically hands the system a concrete task and an evaluation metric chosen by humans, which turns self-evolution into optimizing an explicit objective rather than deciding what and how to learn in the first place.

The paper introduces ASPIRE, a benchmark built around exactly that kind of vague-goal-driven self-evolution. An ASPIRE run gives a system only a natural-language capability goal; the downstream evaluation tasks that will actually be used to grade it stay hidden throughout. The system under test has to operationalize that goal on its own: choose what data to train on and which update method to use, build its own training and validation signals, and decide when it is ready to be evaluated. The environment covers two different ways of self-evolving within the same setup, updating a model's weights directly or editing the agent's harness (the surrounding scaffolding rather than the model itself), and every resulting system is finally scored against a hidden, expert-authored set of 520 items spanning six goals.

Running current agents through ASPIRE, the authors report that vague goals shift a large share of a system's effort toward simply figuring out what the goal means, rather than toward learning itself. Agents do routinely complete full training and harness-editing loops, so the basic mechanics of self-directed learning are not the bottleneck. But weight-level gains from that process are sparse and unstable, and even the strongest self-evolved harness still performs below Qwen-Agent, the paper's engineered reference system. Part of the shortfall traces back to the agents themselves: they often train on data that does not match what they are actually tested on, and they lean on narrow self-evaluations that look convincing locally but do not predict performance on the hidden test set. As a result, apparent local gains frequently fail to transfer, and in some cases continued rounds of search and training erase improvements a system had already made earlier in the process.

Key facts

ASPIRE gives an agent only a natural-language capability goal, such as 'become a better physicist,' while the tasks used to actually evaluate it stay hidden throughout the run.
The system under test has to build its own training and validation signals, choose its own data and update method, and decide for itself when to stop and evaluate.
Final scoring uses a hidden, expert-authored set of 520 items spanning six goals.
Current agents complete training and harness-editing loops routinely, but weight-level gains are sparse and unstable, and the best evolved harness still trails the engineered Qwen-Agent reference.
Agents often train on mismatched data and trust narrow self-evaluations, so local gains fail to transfer to the hidden test, and continued search and training can erase gains made earlier.

Why it matters

Most prior work on LLM self-evolution sidesteps the hardest part of the problem: it hands the system a specific task and an evaluation metric chosen by a human in advance, so 'improving' just means optimizing a known, explicit objective. ASPIRE instead tests whether a system can work out what a vague goal like 'become a better physicist' even means, and how to check its own progress, without ever seeing the actual evaluation criteria, which is closer to how human learners operate when nobody hands them a syllabus.

Who it affects

The benchmark is aimed at anyone building or studying systems meant to improve themselves over time, whether by continuing to train a model's weights or by iterating on the surrounding agent harness. It is also directly relevant to people designing evaluations for self-directed or autonomous learning systems, since ASPIRE specifically isolates the step where a system decides what to learn and how to judge its own progress, rather than just checking whether it can hit a benchmark someone else already defined.

How to use it

As a benchmark, ASPIRE works by handing a system a natural-language capability goal, such as 'become a better physicist' or 'improve at research,' and withholding the tasks that will actually grade it. The system under test then has to operationalize that goal on its own: pick its data and update method, construct its own training and validation signals, and decide when to check its progress. ASPIRE covers two evolution styles in the same environment, direct model-weight updates and agent-harness edits, and produces a final score by running the resulting system against the hidden, expert-authored, 520-item test spanning six goals.

How solid is it

The material available here reads as a paper abstract rather than a full write-up: no individual authors, institution, publication venue, or date appear in the text itself, and no numeric accuracy or pass rate is given for how current agents actually scored on the 520-item evaluation, only qualitative language such as 'sparse and unstable' and 'remains below the engineered Qwen-Agent reference.' The text also does not explain what Qwen-Agent actually is beyond calling it engineered, or exactly how it differs mechanically from the self-evolved harnesses it outperforms, which leaves the comparison directionally clear without a way to gauge from this text alone how large the gap is.

Risks and caveats

The paper's own results double as its caveat: agents often train on data that does not match what they are actually tested on, and they rely on narrow self-evaluations that look fine locally but do not predict the hidden evaluation, so local gains frequently fail to transfer. It also reports that continued rounds of search and training can erase improvements a system had already made earlier, meaning self-evolution under ASPIRE is not guaranteed to move in one direction. On top of that, none of the tested self-evolution approaches yet beats Qwen-Agent, the engineered baseline, so by the paper's own comparison, self-evolving a system does not currently outperform careful engineering on this benchmark.

“Our experiments show that vague goals redirect search effort toward goal interpretation.”

— the ASPIRE paper