Paper: team of AI researchers finds linear attention design that beats GDN2 at 1.3B

A new preprint on Hugging Face Papers asks how a group of AI agents can do scientific discovery together. Autonomous agents can already propose ideas, modify code and run experiments, so the authors want to use teams of collaborative AI researchers to speed up discovery.

Their starting point is that a single search tends to get stuck in local minima. To escape, they propose encouraging orthogonal exploration by building the team from agents with varying expertise. But diverse backgrounds alone are not enough, they argue. Experiments are costly, so a team can test only a limited number of ideas. That forces a balance between exploration and exploitation when choosing which researchers get to propose and run the next experiments. Each result should also shape what the team tries next. The question they pose is how AI researchers with different scientific backgrounds can build on one another's findings and keep making progress under a limited experimental budget.

The design they build has three parts. The researchers are persistent AI agents, each grounded in a different discipline. Each keeps a private research history. All of them share experimental evidence. And research opportunities are handed out through adaptive allocation rather than evenly.

To show it works, the authors deployed the team to search for linear attention architectures. They report that the result outperforms GDN2 at 1.3B scale on language modeling, common-sense reasoning, and synthetic and real-world retrieval tasks.

They also ran controlled comparisons in which one organizational component was removed at a time. Each removal produced markedly weaker discoveries. Without disciplinary backgrounds, the search plateaus early. Without private histories, researchers make less progress even though the shared evidence is identical. With uniform selection instead of adaptive allocation, results fall short within the same budget.

The authors conclude that sustained autonomous discovery depends not only on stronger agents, but also on who explores, what they remember, and how research effort is allocated.

Key facts

The paper builds a team of persistent AI researchers, each grounded in a different discipline, with private research histories, shared experimental evidence and adaptive allocation of research opportunities.
Deployed to search for linear attention architectures, the team reportedly outperforms GDN2 at 1.3B scale on language modeling, common-sense reasoning, and synthetic and real-world retrieval tasks.
Removing any one organizational component yields markedly weaker discoveries in the authors' controlled comparisons.
Without disciplinary backgrounds the search plateaus early; without private histories progress drops despite identical shared evidence; uniform selection trails adaptive allocation on the same budget.
The authors suggest that autonomous discovery depends on who explores, what they remember and how effort is allocated, not only on stronger agents.

Why it matters

Agents that propose ideas, edit code and run experiments are becoming common. This paper argues that how a group of them is organized matters as much as how capable each one is. Its framing is that experiments are costly, so a team can test only a few ideas, and the way those slots are shared out shapes what gets found. Linear attention is the test case: the authors say the team's search produced an architecture that outperforms GDN2 at 1.3B scale on several tasks.

Who it affects

Researchers building automated research or architecture-search systems are the most direct audience, since the paper is a claim about team design rather than about a single model. Groups that run experiments under a fixed compute budget may also care about the finding on adaptive allocation versus uniform selection.

How to use it

The abstract gives the recipe in outline: give each agent a distinct disciplinary background, let each keep a private history of its own research, pool the experimental evidence across the team, and allocate the limited experiment slots adaptively instead of uniformly. The authors' ablations suggest each of these pieces carries weight.

How solid is it

The claims come from the authors' own abstract, and the underlying paper is a preprint. The headline result is a comparison with GDN2 at 1.3B scale. The abstract names no benchmarks and gives no scores for that comparison. The ablations are described as controlled comparisons, and the authors word their conclusion as results that suggest, not prove, their point.

Risks and caveats

The size of the improvement over GDN2 is not stated in the abstract, so "outperforms" cannot be weighed here. The number of agents, the experimental budget and the underlying language models are not given either. The finding is shown on one search problem, linear attention architectures, and the abstract does not say whether it carries over to other domains.

“These results suggest that sustained autonomous discovery depends not only on stronger agents, but also on who explores, what they remember, and how research effort is allocated.”

— From the paper's abstract