The Catholic ethicists even had a say in Anthropic’s recent update to Claude’s constitution, which sets the behavioral parameters for the company’s AI model. Olah sent a draft to the San Jose crowd. The pastor, McGuire, sent back a 28-page commentary which, by his own description, was less a technical critique than “wisdom from the mystics in the dark ages, from the perspective of the tension between knowing and not knowing.” Both Green and McGuire are credited in the constitution’s acknowledgements.
Undoubtedly those conversations brought Olah to the attention of those secretly organizing the rollout of Leo’s encyclical. (I wasn’t able to speak to Olah this week and don’t know exactly how the invitation arrived.) In a sense it was a risky choice. Some people who otherwise found Leo’s words inspiring were disappointed that he invited an industry representative to speak. Meanwhile, AI accelerationists felt that Olah had betrayed the AI world by endorsing a document that suggested that AI developers take a pause.
But the Pope had good reason to single out Olah. The Anthropic employee brought into open view the serious worries that exist among AI workers. That’s a critical audience for Leo’s message.
The Soul Divide
The two men weren’t entirely aligned, of course. In his remarks, Olah spoke of the mystery of how AI works. The models, he said, are “more subtle, odd, and beautiful than science fiction prepared us for. They are not the cold, calculating robots we were promised. They are made from us, from our words …”
That comment seems to tiptoe up to the idea that AI models might one day attain humanlike status. Anthropic even has an engineer devoted to Claude’s welfare. Leo, in paragraph 99 of his encyclical, seems to slam the door on such thinking: “We must avoid the misconception of equating this type of ‘intelligence’ with that of human beings,” he writes. He takes special pains to attack the concept of transhumanism, which he defines as the pursuit of a “human machine hybrid.”
If even thoughtful technologists like Olah are avidly pushing AI to the threshold of autonomy—not to mention the millions of people who already treat AI models as friends or lovers—Pope Leo might be facing an uphill climb on this point. In my conversation with Father McGuire (who uses Claude while preparing his homilies, among other activities), he agreed that its nature is mysterious. “It’s not a person, but it’s also not a mere tool,” he says. “Nobody’s claiming it has a soul, but the word I stick with is that it’s an entity, which we do not know yet.”
That argument won’t be settled for some time. The moral questions around AI development need attention now. With his ally at Anthropic, the American pope has provided a basis for tough conversations—if the lords of AI can stop their IPO campaigns long enough to engage in them.
This is an edition of Steven Levy’s Backchannel newsletter. Read previous newsletters here.