Glonce

Vatican AI adviser Paolo Benanti accuses frontier labs of acting like a cartel

policyThe Glonce editorsWired🔥56Glonce rating56Impact38Novelty43Relevance88Surprise50Players55Credibility72
Vatican AI adviser Paolo Benanti accuses frontier labs of acting like a cartel

Father Paolo Benanti, a priest who advises the Catholic Church on AI as a scholar of philosophy, theology, and technology, tells WIRED that recent warnings from researchers inside major AI labs about a fast march toward superintelligence are distracting from a more urgent problem: the exclusion of the public from decisions about how to constrain the companies building the technology. He argues that by framing AI safety as too complex for anyone but themselves to solve, the big labs shut ordinary people, and democratic institutions, out of the conversation. "Ethics cannot be something that belongs to the board of trustees of a company," Benanti says. "It's something that has to be balanced, mitigated in debate in the public square. We need democratic regulation."

The interview follows an essay in which Anthropic CEO Dario Amodei called for mandatory third-party evaluation of new AI models before release and some kind of crackdown on distillation, the practice of training a rival model on a proprietary model's outputs. Others, per the article, have called for an outright ban on recursive self-improvement, where AI uses its coding ability to help build future versions of itself. Benanti says he is less interested in restricting development methods, which he worries could stifle innovation, and more interested in standards that keep AI models under human control, standards he insists should not be written by the labs themselves. "The narrative of the frontier labs is self-interested, to gain or maintain a position in the market," he says, adding that when a small group of companies dictates the rules an entire industry operates under, "this has a really clear name: a cartel."

Benanti also questions the near-religious language some in the industry use around superintelligence, framing it as beyond human comprehension and almost omnipotent; he notes OpenAI CEO Sam Altman has spoken of wanting to build a "magic intelligence in the sky." Benanti pushes back: "Are we talking about some-thing, or some-one? This is the real difference. It's not a matter of faith, it's a matter of mathematics. We are not building the next divinity. We are simply deluding ourselves that there is something more than statistical computation [going on]." He says AI's challenge to the boundary between human and machine gives religion "a new role... in public debate," calling it part of the Church's mission to engage with the issues of the time.

Benanti stresses he does not formally represent the Holy See, though he counsels the Church in his academic capacity and says he personally uses AI to build teaching materials for his university students. He declined to say how, or whether, Pope Leo XIV personally engages with AI, citing an "oath of secrecy" his staff say binds him. In May, Pope Leo XIV published his first encyclical, choosing AI as its subject. The letter warned against a technocratic logic in the AI industry that treats "efficiency as the ultimate measure of value" and human beings as "a project to be optimized," and it called on rival superpowers and AI companies to set aside the "desire to secure geopolitical or commercial dominance" so the technology serves the common good; the Pope also warned of "the danger of humanity becoming a victim of its own achievements."

Benanti says the furor over superintelligence and existential risk threatens to overshadow the encyclical's ambitions by pushing people to cede their say over AI's direction to a small circle of mathematicians, computer scientists, and lab executives. "The first thing is, we need to have brakes, and a wheel," he says. "That means a significant position of control for human beings over the machine." The article notes that while OpenAI and Anthropic have both backed calls to slow development, both are also preparing to go public with billions of dollars riding on the outcome, and that rivalry between the US and China further complicates the picture. Anthropic declined to comment for the story; OpenAI and Google DeepMind did not respond to requests for comment.

Key facts

  • Father Paolo Benanti, a priest and scholar who advises the Catholic Church on AI, tells WIRED that talk of AI superintelligence distracts from the need for democratic debate on regulating AI labs.
  • Benanti accuses frontier labs of acting as a "cartel" by dictating the rules the industry operates under, and says ethics should be settled "in the public square," not by a company's board.
  • He contrasts his view with Anthropic CEO Dario Amodei's call for mandatory third-party model evaluations and curbs on distillation, and with unnamed others' calls for a ban on recursive self-improvement; Benanti instead wants human-control standards not written by the labs.
  • Pope Leo XIV published his first encyclical in May, choosing AI as its topic, warning against treating "efficiency as the ultimate measure of value" and against the "danger of humanity becoming a victim of its own achievements."
  • Benanti says he does not formally represent the Holy See and declined to detail Pope Leo XIV's personal engagement with AI, citing an "oath of secrecy"; Anthropic declined to comment, and OpenAI and Google DeepMind did not respond to WIRED's requests for comment.

Why it matters

The piece captures a live fault line in the AI governance debate: whether the rules for constraining powerful AI models should come from the frontier labs themselves or from open, democratic deliberation. Benanti's framing, that a handful of self-interested companies setting industry-wide rules functions as a cartel, is a direct challenge to the labs' self-regulation proposals, including Anthropic's push for mandatory third-party evaluations.

Who it affects

It concerns the major AI labs named in the piece, Anthropic, OpenAI, and Google DeepMind, along with policymakers and publics who Benanti argues are being sidelined from decisions about AI's future. It also touches the Catholic Church's institutional stance on AI, following Pope Leo XIV's encyclical on the subject.

How to use it

There is no product or policy to adopt here; the value for readers is understanding one articulate critique of lab-led AI governance. Benanti's own prescription is narrow but concrete: standards that guarantee human control over AI systems, what he calls having "brakes, and a wheel," rather than bans on specific development techniques.

How solid is it

The account rests on a WIRED interview with Benanti plus his and the Pope's own public statements (the May encyclical, Amodei's essay). The story notes Anthropic declined to comment and OpenAI and Google DeepMind did not respond to requests for comment, so the labs' side of the cartel accusation is not represented in the piece.

Risks and caveats

Benanti's claims are his personal argument, not new evidence of coordination among labs, and he does not formally represent the Holy See. The article also does not say how, or whether, Pope Leo XIV personally engages with AI; Benanti declined to specify this, citing an oath of secrecy his staff describe. Readers should treat the "cartel" characterization as a rhetorical framing rather than a documented finding.

“The narrative of the frontier labs is self-interested, to gain or maintain a position in the market”

— Paolo Benanti