Recently, warnings from researchers inside the world’s largest artificial intelligence labs about the rapid pace of development have led to panic about the possibility that the technology could extinguish humanity. Those warnings are predicated on the idea that—propelled by ferocious competition and a rivalry between the US and China—labs are barreling toward superintelligent models whose abilities will far surpass those of humans.
But according to Father Paolo Benanti, a priest who advises the Catholic Church on AI, the handwaving over superintelligence distracts from the need for open debate about how to constrain the companies building the technology. By casting the problem as so complex that only they can solve it, he says, the big labs exclude everyone else from the conversation.
“Ethics cannot be something that belongs to the board of trustees of a company. It’s something that has to be balanced, mitigated in debate in the public square.” Benanti tells WIRED. “We need democratic regulation.”
In an essay published on the heels of the safety warnings, Anthropic CEO Dario Amodei called for new AI models to undergo mandatory third-party evaluations prior to release, and some sort of crackdown on distillation, a process where rival labs train their own AI on the outputs of proprietary models. Others have argued for a total ban on recursive self-improvement, where AI uses its advanced coding capabilities to help develop future iterations of itself.
Benanti is less interested in restricting methods for developing AI, which he fears could quash innovation, and more so in establishing standards that ensure models cannot slip from human control. Whatever shape those standards take, he says, they should not be designed by the labs.
“The narrative of the frontier labs is self-interested, to gain or maintain a position in the market,” says Benanti. When a small group of companies dictates the rules under which an industry operates, he says, “this has a really clear name: a cartel.”
It’s difficult to know how seriously to take grand pronouncements about the capabilities of the latest and future AI models, says Benanti, owing to the overlapping economic, political, and personal incentives at play.
While both OpenAI and Anthropic have supported calls for a slowdown in development, they’re also preparing to go public, and there are many billions of dollars at stake for whoever captures the market. The animosity between the US and China, where the most advanced AI models are being developed, also complicates the picture.
Anthropic declined to comment. OpenAI and Google DeepMind did not respond to requests for comment.
Benanti also takes issue with the way certain corners of the AI industry talk about superintelligence, with the kind of reverence typically reserved for a god: as sophisticated beyond our understanding, practically omnipotent. (OpenAI CEO Sam Altman once said he aspires to build a “magic intelligence in the sky.”) That kind of framing, Benanti says, threatens to skew the discourse about the technology.
“Are we talking about some-thing, or some-one? This is the real difference. It’s not a matter of faith, it’s a matter of mathematics,” he says. “We are not building the next divinity. We are simply deluding ourselves that there is something more than statistical computation [going on].”
By raising questions about what separates man and machine, and how to shield future generations from the negative effects of technology, the advent of AI has the potential to create “a new role for religions in public debate,” says Benanti. “Living the challenge of the time is part of the mission of the Church.”