IEEE Spectrum essay argues superintelligence is doubtful on engineering grounds

The essay opens with a scene from this past summer. Anthropic researcher Jacob Coxon announced his resignation, warning that his employer and OpenAI were "racing straight to self-improving superintelligence and gambling with our lives." Evan Hubinger, Anthropic's head of alignment science, replied that Coxon was correct and that the company really does earnestly believe AI could kill all humans. Anthropic CEO Dario Amodei then told CNN, "I agree with Jacob much more than I disagree with him."
The author explains where those fears come from. The belief is that once engineers build an AI system able to improve itself, it will set off an accelerating cycle of autonomous upgrades, called recursive self-improvement (RSI). The result would be artificial superintelligence (ASI), a system so far beyond humans that it could evade our control and perhaps wipe out our species. The author notes that ASI is a far more ambitious goal than AGI, which generally means AI that matches human abilities across a broad range of cognitive tasks. Reactions so far have varied: some say the ASI concept is used for marketing or to trigger regulation that helps incumbent labs squash new competitors; others debate specific doom scenarios such as AI-engineered superviruses; others hold that aligning ASI with human interests is feasible, so we should rush ahead. The author says one question gets comparatively little attention: from an engineering perspective, is superintelligence even possible?
The author examined this in a computer science doctoral seminar taught at Georgetown University in Washington, D.C., where students studied the technical and cultural roots of the superintelligence story. The class published an annotated bibliography of its readings. The author came away with several objections to the feasibility of ASI, plus a deeper fear that even if the promised intelligence explosion never arrives, pursuing it can still create real harm.
Objection #1 is that RSI is speculative. The idea is old: statistician I.J. Good speculated about an "intelligence explosion" in a 1965 paper, and it gained traction in the 1990s among the Extropians, a Bay Area futurist group, where it let people discuss ASI without covering the engineering. Eliezer Yudkowsky, who later founded the rationalism movement, and Nick Bostrom, who helped steer effective altruism toward existential risk, took it up. Recently it has moved from thought experiment to engineering practice. In a June blog post, "When AI Builds Itself," Anthropic described LLM-powered agents speeding up the writing, debugging and optimization of code and automating parts of research such as running experiments and logging results; its internal data, the report concluded, "is showing that AI is already accelerating the development of AI systems." In a September report, OpenAI announced a goal of developing an automated researcher by March 2028.
The author lists three obstacles. First, data. Through 2023, major LLM advances came from scaling model size: GPT-1 had 117 million parameters, GPT-2 1.5 billion, GPT-3 175 billion, and GPT-4, released three years ago, likely more than a trillion. After GPT-4, continued scaling yielded diminishing returns, so engineers turned to post-training with reinforcement learning. That needs many examples of inputs paired with correct outputs, which are relatively easy to generate in programming and mathematics. That may explain why AI can tackle advanced theorems but struggles to empty an email inbox. The author cites François Chollet's hypothesis that the "jagged frontier" may be mainly math and code, with everything else plateauing because it is bottlenecked by human-generated data, and asserts that automated coding agents and LLM-powered researchers won't overcome this limitation.
Second, scientific effort. Backpropagation did not appear from nowhere: the 1986 paper by David Rumelhart, Geoffrey Hinton and Ronald Williams built on decades of mathematics and computer science. An intelligence explosion would likewise depend on a long sequence of innovations of comparable importance that haven't happened yet, requiring AI agents to simulate, organize and evaluate decades of worldwide scientific effort, which the author calls an impossibly ambitious task. Research led by Princeton computer scientists found current AI tools struggled to make autonomous progress even on highly constrained research questions. The researchers wrote that their results "provide early evidence that today's agents can do the engineering of AI research, but struggle with critical parts of the research life cycle."
Third, resources. Tens of thousands of human brains have worked for decades, aided by increasingly powerful computers, to build today's AI. The author asks what happens if each next intelligence level needs double the resources of the last; soon, no number of chips or data centers could supply the compute. The author concludes that RSI leading to an intelligence explosion is highly speculative and rests on many unproven assumptions. RSI is not definitively impossible, but it deserves more technical skepticism than it gets, and Carl Sagan's adage about extraordinary claims needing extraordinary evidence applies.
Objection #2 is that superintelligence might not be computable. The author says superintelligence talk implicitly assumes computers can perform any imaginable calculation given a smart enough algorithm and powerful enough processors, with RSI as the process that finds those algorithms. Computer scientists do not think this way. In his 1937 paper "On Computable Numbers," Alan Turing proved that most problems cannot be solved by algorithms, because the space of possible problems is uncountably infinite while the space of algorithms is only countably infinite. Computational complexity theory later showed that many of the problems computers can technically solve would need an astronomically large amount of time and resources, making them practically impossible. So, the author says, we cannot simply assume that any science fiction scenario about powerful AI is possible if machines get enough time to self-improve. The account here runs to the opening of this second objection.
Key facts
- The author, who taught a computer science doctoral seminar at Georgetown University, argues that RSI leading to an intelligence explosion is highly speculative and rests on many unproven assumptions, though not definitively impossible.
- Objection #1 has three parts: reinforcement-learning post-training needs data that is easy to get only in fields like coding and math, an explosion would need many breakthroughs comparable to backpropagation, and each intelligence level might need double the resources of the last.
- Objection #2 is that superintelligence might not be computable: Turing's 1937 paper showed most problems cannot be solved by algorithms, and many solvable ones would take astronomically large time and resources.
- The essay starts from Anthropic's summer debate: Jacob Coxon resigned warning of a race to self-improving superintelligence, Evan Hubinger agreed, and Dario Amodei told CNN he agreed with Coxon much more than he disagreed.
- Anthropic's June post "When AI Builds Itself" says AI is already accelerating AI development, and OpenAI announced a September goal of an automated researcher by March 2028; a Princeton-led study found agents struggle with critical parts of the research life cycle.
Why it matters
Debate over superintelligence has mostly focused on marketing motives, specific doom scenarios and alignment. The author says the narrower question of whether ASI is feasible as engineering gets comparatively little attention, and that the answer can help calibrate anxieties and shape better responses. The essay also sets the argument against concrete company moves: Anthropic reports AI already accelerating AI development, and OpenAI has announced a goal of an automated researcher by March 2028.
Who it affects
Anyone following AI safety and policy arguments, and the labs themselves: the essay names Anthropic and OpenAI as organisations pursuing automated research. It also speaks to people weighing how seriously to take extinction warnings from lab insiders such as Coxon, Hubinger and Amodei. The author's deeper fear is that pursuing an intelligence explosion that never arrives can still create real harm.
How to use it
Treat it as a framework for questioning claims about recursive self-improvement. Ask whether the training data exists for the domain in question, whether the needed scientific breakthroughs have actually happened, how compute needs would scale from one level to the next, and whether the problem is computable at all. The essay also ends with an infographic of three possible futures: an intelligence explosion; danger without superintelligence, where increasingly autonomous AI causes harm through misuse, poor design or loss of human control; and a useful AI future of powerful, specialized tools with constraints that make them easier to predict and control.
How solid is it
This is an opinion essay, and the author frames the conclusions as personal belief ("I believe..."). Some supporting material is checkable and attributed: Anthropic's June post, OpenAI's September report, the Turing and backpropagation papers, and a Princeton-led study. Others are tentative by the author's own wording: the data explanation "might" account for AI's uneven skills, and the doubling of resources per level is posed as a what-if, not a finding. The GPT-4 parameter count is marked "likely" more than a trillion. The author does not claim superintelligence is proven impossible. The retelling here covers the essay through the start of Objection #2, so later objections and the full argument about harm are not covered.
Risks and caveats
The essay argues that RSI is not definitively impossible, so it is a call for skepticism, not a refutation. It offers no timescale for when, or whether, ASI might arrive. The data argument leans on the author's assertion that coding agents and LLM-powered researchers won't overcome the data limitation, while Anthropic reports the opposite trend inside its own labs. The Princeton finding is mixed: agents can do the engineering of AI research but struggle with critical parts of the research life cycle. The computability argument, as retold here, is only begun, so its full strength cannot be judged.
“We cannot simply assume that any science fiction scenario we conjure about powerful AI is possible if we give machines enough time to self-improve.”
— Essay author, who taught a computer science doctoral seminar at Georgetown University, IEEE Spectrum