Continual-ARC benchmark finds agent harnesses gain nothing on unseen tasks

The paper proposes Continual-ARC, a benchmark for continual learning built from the three ARC-AGI editions. Its starting point is that agent harnesses have emerged as a promising way to learn continually, alongside test-time parametric learning (updating a model's weights while it works). The authors say current benchmarks fall short in three ways: they generally focus on a single applied domain, serve each instance only once, and assume a single family of learner. As a result, memory-based and weight-based systems are rarely compared on the same scale.
Continual-ARC is a lifelong stream of tasks that need no domain knowledge. The same tasks come back after dormant gaps, each time with fresh instances, while new tasks keep arriving. On each instance the learner has options: it may pay for demonstrations, submit answers, or act inside the harness. Retries are permitted on incorrect submissions, but they are paid for. Each instance is scored by the share of this feedback budget the learner consumed. Because a learner may keep anything between instances, memory-based and weight-based systems end up scored on the same scale.
After the stream ends, the learner is frozen and tested twice: on fresh instances of tasks it met, which measures memorization, and on unseen tasks, which measures generalization.
For experiments, the authors run an agent harness with frozen weights, with and without persistence, on two base models, Qwen3.5-9B and GLM-5.3 Flash. They compare these runs with two prompting floors, a program-induction method, and Qwen3.5-9B's weights trained to imitate GLM-5.3 Flash.
Three findings come out of this. Persistence in the harness generally does not improve memorization unless it is provided with explicit expert guidance. Most harness-based methods still learn less than a floor that simply caches each task's demonstrations, across all models tested. And no harness-based method gains on unseen tasks, whereas weight-based learning shows substantial gains. The authors conclude that this opens the door to future weight-based continual learning that could teach models to drive harnesses in new domains.
Key facts
- Continual-ARC is a lifelong benchmark stream built from the three ARC-AGI editions; tasks need no domain knowledge and return after dormant gaps with fresh instances while new tasks keep arriving.
- Each instance is scored by the share of a feedback budget consumed: the learner can pay for demonstrations, submit answers, or act in the harness, and retries on wrong answers are allowed but paid for.
- After the stream, the learner is frozen and tested on fresh instances of tasks it met (memorization) and on unseen tasks (generalization).
- Most harness-based methods learned less than a floor that simply caches each task's demonstrations, across all models tested, and persistence helped memorization only with explicit expert guidance.
- No harness-based method gained on unseen tasks, while weight-based learning, including Qwen3.5-9B trained to imitate GLM-5.3 Flash, showed substantial gains.
Why it matters
Agent harnesses, where an agent keeps notes or memory between tasks, have emerged as a promising mechanism for continual learning. Weight-based learning is the other route. Until now the two have rarely been compared on one scale, because existing benchmarks usually cover a single applied domain, serve each instance once, and assume one family of learner. Continual-ARC is an attempt to put both families on the same yardstick, using tasks that need no domain knowledge. The authors' early result is a sobering one for harnesses: they did not gain on unseen tasks.
Who it affects
Researchers building continual-learning methods and agent frameworks are the direct audience, since the benchmark is meant to score memory-based and weight-based systems together. Teams that rely on harness persistence (memory carried between tasks) to make agents improve over time are also affected, because the authors found that persistence generally did not improve memorization without explicit expert guidance.
How to use it
The paper offers a protocol more than a product. A learner is run through the stream, paying a feedback budget on each instance, then frozen and tested on fresh instances of known tasks and on unseen ones. A practical baseline from the paper is the floor that simply caches each task's demonstrations: a harness method that cannot beat it is learning little. The authors also suggest a direction: weight-based continual learning that teaches models to drive harnesses in new domains.
How solid is it
This is a preprint, and the source text is its abstract. Its findings are the authors' own claims from their own tests on two base models, Qwen3.5-9B and GLM-5.3 Flash. No authors or institutions are named. No numerical scores, accuracy figures, or effect sizes are given for any method. The size of the stream (number of tasks or instances) and the length of the dormant gaps are not stated. The abstract does not say which harness-based methods, if any, beat the caching floor. No release of code or data is mentioned.
Risks and caveats
The headline claim, that no harness-based method gains on unseen tasks, comes from a limited set of tested configurations, so it should not be read as a verdict on all harness designs. The abstract does not say how much expert guidance was given or what form it took, which matters because guidance is the one condition under which persistence generally helped memorization. The weight-based result also rests on one specific setup: Qwen3.5-9B's weights trained to imitate GLM-5.3 Flash. Without published figures, the size of the gap between the two families cannot be judged.
“No harness-based method gains on unseen tasks, whereas weight-based learning shows substantial gains.”
— Continual-ARC paper abstract