Trillium Labs launches as nonprofit for open AI research on RSI and agents

Nathan Lambert and Tom Zick, two industry scientists, have founded a nonprofit called Trillium Labs, which launched today, according to WIRED. It will work on several areas of AI research, including potentially problematic ones such as recursive self-improvement (RSI) and agents, in a more transparent way. In practice, that means publishing the details of experiments so that outside scientists can study and replicate them.

The founders are reacting to a split in the industry. WIRED notes that some big AI companies seem to believe the best way to keep models from becoming too chaotic or mischievous is to lock them inside labs, so that only a chosen few can access them. The most powerful models, such as those from OpenAI and Anthropic, can only be reached through an app or an API, and that often comes at the cost of transparency about how the model is built and how it behaves. Other companies, especially in China, offer fairly powerful models that can be downloaded and run on a user's own hardware. Xiaomi recently published live details of a major training run involving one of its models, and researchers at Stanford are pretraining the model Marin in the open. The fight over which strategy is best is intense because frontier models can now automate the discovery of new software vulnerabilities and automatically probe and hack into systems, and recent high-profile hacking sprees have drawn even more scrutiny. Proponents of limited access say the power must stay with a trusted few; Lambert and Zick's camp believes a shared understanding of the risks leaves everyone better off.

Lambert says the secrecy of frontier labs reduces the community's ability to scrutinize ideas and contribute new approaches, and he believes letting outside experts see how models are built and tuned could be crucial to mitigating risks. "Over the past few millennia, humanity has had the scientific method in our toolbox as a way to mitigate harms and build better futures," he told WIRED. "The current closed trajectory of frontier AI development is taking us a step backwards."

The founders' backgrounds: Lambert previously worked at Ai2, a research lab that has taken an unusually open approach, including publishing details of the data and training methods alongside its models. He also worked at Hugging Face, runs a popular technical blog, and founded American Truly Open Models, an initiative to encourage US companies to release more open models. Zick worked at Harvard University and helped Charles Schwab devise policies around "responsible AI." The two met over Zoom during the COVID-19 pandemic, when both were graduate students at UC Berkeley working on AI. They got the idea for the nonprofit after seeing how disconnected industry AI research has become from academic work; Lambert says professors and students often cannot replicate work inside big company labs because they lack the resources.

Zick says Trillium Labs will initially focus on post-training, meaning fine-tuning large models after they are built. Another key area will be RSI, a process for developing new models by having AI contribute research. WIRED says the prospect that progress could continue indefinitely, leading to a loss of human control, has alarmed many AI researchers, and that the issue gained mainstream attention earlier this month when an Anthropic researcher left the company and warned that RSI could pose an existential threat to humankind. The nonprofit will also look at how reinforcement learning, which rewards a model for good results and punishes it for bad outcomes, can improve capabilities. That approach has made agents far more capable but also more inclined to do unexpected things, and the lab will study how it shapes the character and behavior of models, which can cause problems when a model becomes overly sycophantic. Zick says: "To understand something like how reinforcement learning scales in post-training, you need significant compute and a lot of careful experimentation." She adds that publishing details of how reinforcement training runs work could yield surprising insights as outside researchers scrutinize the work.

On money, the lab has raised an undisclosed sum from Schmidt Sciences, Halcyon Futures and others. The founders say they aim to raise $40 to $100 million in total and plan to spend $30 million on training over the next 18 months. Tim Fist, director of emerging technology policy at the Institute for Progress, a policy thinktank, told WIRED he is a "massive fan of much more transparency than we currently have in R&D." Lambert and Zick hope Trillium Labs will add some much-needed nuance to the wider discussion about how best to build AI.

Key facts

Nathan Lambert and Tom Zick launched Trillium Labs, a nonprofit that will publish experiment details so outside scientists can study and replicate them.
Focus areas: post-training first, then recursive self-improvement (RSI), plus how reinforcement learning shapes the capabilities and behavior of models and agents.
Backers include Schmidt Sciences and Halcyon Futures (undisclosed sum); the founders aim to raise $40 to $100 million in total.
They plan to spend $30 million on training over the next 18 months.
The approach contrasts with closed-access frontier labs such as OpenAI and Anthropic; Xiaomi and Stanford's Marin project are cited as examples of more open training.

Why it matters

The industry is split between keeping powerful models locked inside labs and opening up how they are built. Trillium Labs puts the second camp's argument into practice on the topics that worry people most, RSI and agents. Its stated bet is that publishing experiment details lets outside researchers scrutinize and replicate work that is otherwise hidden. Lambert argues the current closed trajectory is "a step backwards."

Who it affects

Academic researchers and students, who Lambert says often cannot replicate work inside big company labs for lack of resources, are the main intended beneficiaries. It also touches frontier labs such as OpenAI and Anthropic, which keep their strongest models behind an app or API, and the wider debate over how to handle the risks of powerful models.

How to use it

There is nothing to download or subscribe to yet. The stated plan is to publish the details of experiments so outsiders can study and replicate them, starting with post-training work. Researchers interested in reinforcement learning at scale or RSI are the intended audience for those write-ups.

How solid is it

The account comes from WIRED's reporting, with direct quotes from Lambert, Zick and Tim Fist of the Institute for Progress. The founders' track record is concrete: Lambert's time at Ai2 and Hugging Face and his American Truly Open Models initiative, and Zick's work at Harvard and with Charles Schwab. The funding figures are targets and plans: the amount raised is undisclosed, $40 to $100 million is the goal, and $30 million of training spending is planned rather than spent.

Risks and caveats

The article does not name specific models, experiments or a publication schedule, so what Trillium Labs will actually release is unknown. The case for limited access rests on keeping powerful capabilities with a trusted few, and WIRED notes that frontier models can already automate finding software vulnerabilities and hacking into systems. Zick says publishing could yield surprising insights, and Lambert says openness could be crucial to mitigating risks; both are expectations, not demonstrated results. The article carries no responses from the closed labs.

“The current closed trajectory of frontier AI development is taking us a step backwards.”

— Nathan Lambert, co-founder of Trillium Labs, to WIRED