Anil Seth's prize-winning essay challenges the case for conscious AI

Noema Magazine has published "The Mythology Of Conscious AI", the essay by Anil Seth that won the 2025 Berggruen Prize Essay Competition. Seth is a neuroscience professor and director of the Centre for Consciousness Science at the University of Sussex, and co-director of the Canadian Institute for Advanced Research Program on Brain, Mind and Consciousness. He is the author of the 2021 book "Being You: A New Science of Consciousness".
Seth opens with the long cultural history of people trying to play God by building artificial humans, from Yossele the Golem and Frankenstein to HAL 9000, Ava in "Ex Machina" and Klara in "Klara and The Sun". Those stories rarely end well, he notes, and they suggest that a path toward real consciousness, or toward systems that persuasively seem conscious, puts a lot at stake beyond job markets.
He then sets out why the question is live. In a 2022 Washington Post interview, Google engineer Blake Lemoine claimed that a chatbot called LaMDA was conscious, had feelings and was in an important sense like a real person. He was not taken very seriously, and Google dismissed him for violating its confidentiality policies. Seth points out that firing someone for breaching confidentiality is not the same as firing them for being wrong. David Chalmers has suggested conscious machines may be possible in the not-too-distant future, and Geoffrey Hinton thinks they exist already. In late 2024 a group of prominent researchers wrote a widely publicized article on the need to take the welfare of AI systems seriously. For many leading experts in AI and neuroscience, Seth says, machine consciousness is a question of when, not if.
Why it matters, in his account: if AI systems are conscious, they would have moral status, the potential for suffering and perhaps rights. It matters for people too, whatever the reality. If we feel our AI companions really feel things, our psychological vulnerabilities can be exploited and our ethical priorities distorted, and treating conscious-seeming machines as if they lack feelings is psychologically unhealthy. If we grant AI creations rights, we may not be able to turn them off even when they act against our interests. And if we confuse ourselves too readily with our machines, we overestimate them and underestimate ourselves.
The core of the visible argument is a section on the temptations of conscious AI. Seth says intelligence and consciousness are different things. Intelligence is mainly about doing; he offers a general definition of it as the ability to achieve complex goals by flexible means. Consciousness is mostly about being. He cites Thomas Nagel's formulation that an organism has conscious mental states if and only if there is something it is like to be that organism. Consciousness is the difference between wakefulness and the oblivion of deep anesthesia, the colors, tastes and emotions that give life texture. AI systems can reasonably claim intelligence in some form, but it is harder to say whether there is anything it is like to be ChatGPT.
Seth traces the habit of bundling the two to three baked-in psychological biases. Anthropocentrism takes the human case as definitional rather than as one example of how properties can combine. Human exceptionalism puts our species at the top of every pile. Anthropomorphism projects humanlike qualities onto nonhuman things based on superficial similarities. Language is especially effective at triggering these biases, which is why LLMs such as OpenAI's ChatGPT and Anthropic's Claude draw most of the excitement. As far as Seth knows, nobody has claimed DeepMind's AlphaFold is conscious, even though under the hood it is rather similar to an LLM; it predicts protein structure rather than words and does not pull our psychological strings in the same way.
Vocabulary matters too. Calling LLM falsehoods "hallucinations" implicitly gives them a capacity for experience, since human hallucinations are conscious experiences that have lost their grip on reality. If a human analogy is needed, he says, "confabulate" is far better: it means making things up without realizing it, which is about doing rather than experiencing. He adds that we should not assume consciousness will come along for the ride as AI gets smarter, and that anyone who says it will emerge at the arbitrary threshold of AGI is showing human exceptionalism at work.
Seth names further biases. One is the sense that everything in AI is changing exponentially, which makes us feel always on the cusp of a major transition; but on an exponential curve every point is an inflection point. Another is the techno-rapture: the Promethean lure of bringing to humankind what was once the province of the divine, with the promise of immortality in a silicon cloud. He quotes Caleb in "Ex Machina": "If you've created a conscious machine, it's not the history of man, that's the history of Gods." He suggests this may be why claims of imminent machine consciousness seem more common among the technorati. In a more cynical aside he adds that fueling the idea of something semi-magical about AI may help share prices stay aloft and justify Silicon Valley's salaries and investment.
He also cites Adam Becker's book "More Everything Forever", which describes projecting consciousness into AI as a form of pareidolia, like seeing a face in a piece of toast. Seth calls this apt, but says recognizing these biases is only the first step. To ask whether real artificial consciousness is even possible, he says, we need to dig deeper, and the next section begins with the assumption that consciousness is a matter of computation. The available text stops there.
Key facts
- Anil Seth, a neuroscience professor and director of the Centre for Consciousness Science at the University of Sussex, won the 2025 Berggruen Prize Essay Competition with this essay, published by Noema.
- Seth says intelligence (mainly about doing) and consciousness (mostly about being) are different things, and that we should not assume consciousness will come along as AI gets smarter.
- He traces the urge to bundle the two to three biases: anthropocentrism, human exceptionalism and anthropomorphism. He adds exponential-growth thinking and the techno-rapture as further temptations.
- He argues that "confabulate" is a better word than "hallucinate" for LLM falsehoods, because hallucination implies conscious experience.
- He says the idea of conscious AI rests on the assumption that consciousness is a matter of computation, and promises to dig deeper into whether artificial consciousness is possible.
Why it matters
Seth argues that beliefs about machine consciousness have consequences whether or not they are true. If AI systems were conscious, they would carry moral status, the potential for suffering and perhaps rights. If people only believe their AI companions feel things, their vulnerabilities can be exploited and their ethical priorities distorted. He also warns that granting rights to AI could leave us unable to turn systems off, even when they act against our interests. The essay frames the debate against Lemoine's 2022 LaMDA claim, Chalmers's suggestion that conscious machines may come in the not-too-distant future, Hinton's view that they exist already, and a late-2024 article by prominent researchers on AI welfare.
Who it affects
The essay speaks to AI researchers, neuroscientists and philosophers who treat machine consciousness as a question of when, not if. It also touches ordinary users of chatbots and AI companions, who may be inclined to see feelings in systems that talk fluently, and the companies that build and market those systems. Seth also says that confusing ourselves with our machines makes us overestimate them and underestimate ourselves.
How to use it
The piece offers some practical habits of thought. Separate what a system does (intelligence) from whether there is something it is like to be that system (consciousness). Notice the pull of anthropocentrism, human exceptionalism and anthropomorphism, especially when a system uses language. Be careful with loaded words: Seth prefers "confabulate" to "hallucinate" for LLM errors. And treat claims that consciousness will emerge at the AGI threshold with suspicion.
How solid is it
This is a philosophical opinion essay by a neuroscientist, and it won a major essay prize; it is not an experiment or a dataset. It gives no survey or figure on how many experts believe in machine consciousness. The text available ends at the start of the section on consciousness and computation, so Seth's argument on whether consciousness is computation and his final conclusions are not covered here. The visible text does not say Seth believes AI cannot be conscious; it challenges the assumption that consciousness comes with intelligence and says the question of possibility needs deeper examination.
Risks and caveats
The bias arguments explain why people might over-attribute consciousness, but on their own they do not settle whether AI could be conscious; Seth himself says that question needs digging deeper. His remark that the idea of semi-magical AI may help share prices stay aloft is a cynical aside in parentheses, not a demonstrated claim. The Lemoine episode is told from Seth's side: he says Google dismissed Lemoine over confidentiality, which is not the same as judging him wrong. Claims about what Chalmers and Hinton think, and about Becker's pareidolia argument, are Seth's descriptions of their views.
“Intelligence and consciousness are different things.”
— Anil Seth, "The Mythology Of Conscious AI", Noema