Richard Sutton says raw compute alone is not enough, stirring the LLM scaling debate

Richard Sutton, a Turing Award winner and the author of the influential 2019 essay "The Bitter Lesson," wrote in a recent post that raw compute alone is not enough for intelligence. AI science, he says, needs clever new methods and is "far from finished." The Decoder reports that the post sparked wide discussion in the AI community on X, and reads that as a sign Sutton struck a nerve.

The irony is the point of the headline. In the 2019 essay, Sutton argued that general methods which scale with compute eventually beat sophisticated human-engineered approaches; what matters are ideas that grow with compute rather than rely on human assumptions. The Decoder stresses that Sutton's current position is not new. In earlier statements he criticised the limits of simply scaling large language models and called synthetic data a "big mistake," because the real world is infinitely complex and any simulation of it is only "microscopic." He also said human expertise as a bottleneck prevents true scaling. Instead he called for agents with world models that continuously learn from their own experience. The world is massively more complex than what internet text can capture, he argues, and today's language models do not keep learning after training. What is needed is continuous learning without destroying old knowledge.

Two researchers responded. Jerry Tworek, an AI researcher who, like many others, left OpenAI to explore new approaches, says the bitter lesson still holds, but the more compute goes into training, the more the algorithm matters. Some algorithms scale, others hit dead ends, and many dimensions of scaling remain completely unexplored. In his words, the temporary success of transformer scaling, combined with a misreading of the bitter lesson, "has reward-hacked the brains of a whole generation of scientists."

Jayden Teoh, another AI researcher, says he has repeatedly seen the bitter lesson hold back academic research, with promising ideas dismissed for straying too far from its spirit. He argues this is not Sutton's fault but a misreading of the original message. Ideas that in hindsight did not follow the bitter lesson were often replaced by cleaner, more general approaches, Teoh says, but that did not make them wrong; they were often what enabled progress at the time. He wants the community to stop dismissing such work as soon as it appears.

The Decoder ties this tension to a growing wave of startups: in its Frontier Radar #5 it tracked 45 new AI labs betting on these gaps. Tworek runs Core Automation, one of them, searching for learning algorithms beyond pretraining and reinforcement learning. Sutton is named among the founders involved, with Oak Lab. What unites them, the article says, is belief in much more capable AI but doubt that today's recipe alone will get there.

Industry leaders see it differently. OpenAI CEO Sam Altman considers LLM scaling far from exhausted and accuses critics of slowing down an entire generation of researchers by being too sure about what scaling cannot do. As evidence he points to an OpenAI model that recently disproved a mathematical conjecture mathematicians had long failed to crack. He finds world models useful, for instance in robotics, but says the data clearly favor further LLM scaling. Anthropic CEO Dario Amodei argues similarly, calling scaling progress a rainbow with no end. The Decoder adds its own assessment: as heads of the two companies whose entire technical approach rests on this bet, Amodei and Altman are obviously biased in its favor.

Key facts

• Richard Sutton, Turing Award winner and author of the 2019 essay "The Bitter Lesson," says in a recent post that raw compute alone is not enough for intelligence and that AI science is "far from finished."
• His earlier statements called synthetic data a "big mistake" and urged agents with world models that learn continuously from their own experience without destroying old knowledge.
• Jerry Tworek says the bitter lesson holds but the algorithm matters more as compute grows; Jayden Teoh says the lesson has been misread to dismiss promising academic ideas.
• Sam Altman and Dario Amodei still back LLM scaling; Altman cites an OpenAI model that disproved a long-open mathematical conjecture.
• The Decoder tracked 45 new AI labs betting on gaps in current LLM scaling in its Frontier Radar #5, including Tworek's Core Automation and Sutton's Oak Lab.

Why it matters

The author of the best-known argument for scaling with compute is now saying compute is not the whole story. That matters because "The Bitter Lesson" is often cited to justify scaling bets. Tworek and Teoh both argue the essay has been misread, which turns a slogan into a live dispute about what it actually says. Behind it sits a practical question: whether today's recipe alone leads to much more capable AI, or whether new learning methods, such as continuous learning, are required.

Who it affects

Researchers choosing between scaling existing LLMs and trying new learning algorithms; academics whose less conventional ideas, Teoh says, get dismissed for straying from the bitter lesson; founders of the new labs The Decoder tracks, including Core Automation and Oak Lab; and the two companies built on the scaling bet, OpenAI under Sam Altman and Anthropic under Dario Amodei.

How to use it

This is a debate, not a product or technique. It is useful as a map of positions: Sutton wants continuously learning agents with world models; Tworek says algorithms matter more as compute grows; Teoh asks that unconventional ideas not be written off because they may not survive scaling; Altman and Amodei say the evidence still favors scaling LLMs. Readers who want the original arguments should look to Sutton's post and the responses on X; the article gives no date or venue for the post.

How solid is it

This is opinion, reported secondhand by The Decoder, with no new data or experiment. Sutton's critiques of scaling are described as repeats of earlier statements. Altman's evidence is an unnamed OpenAI model and an unnamed conjecture, and the article gives no date or source for his or Amodei's remarks. The "rainbow with no end" phrase is the article's paraphrase, not a quotation. The claim that the post sparked wide discussion on X is The Decoder's own characterisation.

Risks and caveats

Both camps argue from positions: the article itself says Altman and Amodei are obviously biased toward scaling, and Tworek and Sutton are involved with labs that bet on alternatives. The Decoder's count of 45 labs comes from its own tracking and does not show which approach will work. Each side's position is also compressed here, so the full posts may carry more nuance than the summary.

“An idea doesn’t necessarily have to survive scaling forever to be an important contribution to AI progress”

— Jayden Teoh, AI researcher, as quoted by The Decoder