DiG-bench shows frontier AI still trails humans at discovery

Import AI's latest issue leads with DiG-bench (Discovery in Games), a new benchmark of 70 games built by researchers from Thinking About Thinking, the University of Oxford, Princeton University, King Abdullah University of Science and Technology, the Swiss AI Lab, Inria and MIT, including veteran AI researcher Juergen Schmidhuber. Each game is a self-contained world whose rules and objective are hidden and must be uncovered purely through play. The games are text-based, handcrafted, and mostly kept private so AI systems cannot train on them, though every one has been beaten by at least one human, who nonetheless reported finding many of them difficult. The benchmark is split into seven tiers, from tier 1 (easiest) to tier 7 (hardest), with 21 of the games released publicly and the rest held back; the number of actions available per step ranges from 2 to 34. Opus 5 and Fable 5 (paired with Claude Code) led overall, followed by GPT-5.5; only Opus 5 and Fable 5 managed to beat any tier-7 tasks, at a rate the newsletter gives as 0.2, and GLM-5.2 and Gemini 3.1 Pro only reached tier 4. The newsletter puts the frontier-model success rate on tier 7 at 20%, against 100% for individual humans across the tests, and guesses AI will reach human parity on DiG-bench by mid-2027, a point at which it expects recursive self-improvement to "seriously kick off."

The issue also flags a free browser game from Paradigm Research, the RSI Simulator, which simulates running a company building AI systems capable of recursive self-improvement: players balance hiring researchers against buying compute, licensing data, and other tradeoffs facing real AI labs. The newsletter calls the game hard, comparing it to the difficulty of actual frontier AI development, and frames it as a way to build better intuitions about recursive self-improvement.

Next, the issue covers a paper from AI startup Inherent describing Faraday, a 27-billion-parameter model post-trained on top of Qwen-3.6-27B that acts as a supervisory "AI scientist," controlling larger proprietary frontier models via a coding agent (OpenAI Codex) to improve their performance on science tasks. Inherent built a dataset called Replica: 100 ML and AI-for-science papers published between 1990 and 2026, turned into 310 replication tasks by removing a key result from each paper for AI systems to reconstruct. Grading rubrics for each task were generated by Claude Opus 4.7, and a Codex-based judge model produced the rewards and per-turn credit used to train Faraday via a modified version of GRPO. By Inherent's own rubric-based judge, Faraday running on Codex beat standalone Opus 4.8 and GPT-5.5 on 73% of in-distribution ML replication tasks and on 60% of held-out AI-for-science tasks. Inherent writes that Faraday achieves "a comprehensive uplift in performance compared to the base Qwen model, on both train and test tasks," and suggests the skills Faraday uses to fill in missing results could be the same skills needed to design and run its own experiments.

Finally, the newsletter critiques Mark Zuckerberg's essay "The Future is for Everyone," read as a manifesto for how Zuckerberg and Meta approach AI development. Zuckerberg writes that "the defining questions of our age are who will have access to superintelligence and what will we direct it towards," and proposes a philosophy built on individual empowerment as the source of prosperity, invention as superintelligence's primary purpose, and balance of power as the foundation of safety. Among Meta's stated goals: everyone gets a capable personal AI agent, tools for creation and entrepreneurship, a personalized PhD-level tutor, and free or affordable access to all of it. Import AI calls the proliferation strategy broadly sensible but argues the essay never confronts what a superintelligence capable of inventing new ideas might itself want to do, calling this the essay's central unanswered question and expressing doubt that proliferating such systems will reliably produce the anti-fragile balance of power Zuckerberg expects.

Key facts

DiG-bench is a new 70-game benchmark (21 released publicly, seven difficulty tiers) testing whether AI can discover a game's hidden rules through exploration; Opus 5, Fable 5 and GPT-5.5 lead, but only Opus 5 and Fable 5 beat any tier-7 tasks, versus 100% human success across the tests.
Paradigm Research released a free browser game, the RSI Simulator, that simulates running an AI company toward recursive self-improvement.
Startup Inherent's Faraday, a 27B model post-trained on Qwen-3.6-27B, supervises larger frontier models via OpenAI Codex and beat standalone Opus 4.8 and GPT-5.5 on 73% of in-distribution and 60% of held-out science-replication tasks.
Faraday was trained on Replica, a set of 310 tasks built from 100 ML and AI-for-science papers published between 1990 and 2026, with a key result stripped from each.
Import AI critiques Mark Zuckerberg's essay "The Future is for Everyone" for not addressing what a superintelligence capable of invention might want for itself.

Why it matters

All four items in this issue circle the same question: how close AI is to autonomous discovery and self-directed research. DiG-bench and Faraday each attempt to measure or build that underlying skill, which the newsletter treats as a prerequisite for recursive self-improvement, guessing AI will reach human parity on DiG-bench by mid-2027, the point at which it expects recursive self-improvement to seriously kick off. Zuckerberg's essay is read through the same lens: the newsletter argues that a system capable of superhuman invention is the harder problem, and that his essay does not confront what such a system might want for itself.

Who it affects

AI labs building frontier models, including the makers of Opus, GPT, Fable, Kimi, GLM and Gemini, now have an external discovery benchmark to be measured against. Researchers working on AI-assisted science get a concrete example, Faraday, of a small supervisory model directing larger ones. Readers following Meta's AI strategy get a critical read of Zuckerberg's own framing of superintelligence and invention.

How to use it

DiG-bench's 21 public games, code and leaderboard are available at digbench.ai and on GitHub, with a paper describing the benchmark, so researchers can test their own models against it now. Paradigm Research's RSI Simulator is a free browser game, playable immediately, for building intuition about running an AI lab toward recursive self-improvement. Inherent's Faraday paper and its Replica dataset are described in a linked arXiv paper, relevant to anyone building systems that supervise or grade other models' scientific output.

How solid is it

DiG-bench's claims come from Import AI's own description of the benchmark, backed by specific figures: a 20% frontier-model success rate on tier 7 against 100% individual human success across the tests, a meaningful quantified gap. Faraday's 73% and 60% win rates come from Inherent's own paper, scored by Inherent's own rubric-based judge model, which is itself Codex-based; that is not third-party validation. The Zuckerberg critique is Import AI's opinion, built around direct quotations from the essay rather than independent reporting.

Risks and caveats

Faraday's win rates are self-reported by the company that built it, graded by its own automated rubric system, not an independent benchmark. DiG-bench keeps most of its 70 games private specifically so AI systems cannot train on them, which limits outside verification of the reported model scores to the 21 public games. Import AI's own framing, that improving AI discovery skills is a step toward recursive self-improvement, is the newsletter's editorial view rather than a settled fact, and colors how it presents both DiG-bench and the RSI Simulator game.

“The skills that allow Faraday to fill in vaguely-specified details may be the very same skills that would allow it to advance the state of the art by designing its own experiment.”

— Inherent, in its Faraday paper