The researchers note, however, that Anthropic’s 97 percent detection rate likely applies to full scam conversations including the appeal for a fake investment, not the relationship-building phase they focused on, which includes far less conspicuous language. Their findings about Claude’s willingness in many cases to impersonate a human, the researchers note, were carried out with the latest version of Anthropic’s chatbot.
The efficiency of using AI for scam relationship-building contrasts with the researchers’ findings from their broad survey of scam workers, which found that they use LLMs only as a supplementary tool for refining their scams. The reason for that disconnect, the researchers concluded, is that human trafficking may still be cheaper and more profitable than AI for scam organizations. That’s not only because it’s nearly free labor—given compounds often enslave workers without pay or pay them only a small income that keeps them in debt bondage—but also because the same human trafficking victims are sometimes ransomed for payment at the end of their time in a scam compound. “It may be that they don’t yet feel forced to automate,” says Mirsky of Ben Gurion University of the Negev. “It’s not only free labor, they also get money for those people as well.”
The study’s results suggest, nonetheless, that the scam industry may soon shift to more AI automation, says Erin West, a former Santa Clara County, California, prosecutor who now leads an anti-scam organization called Operation Shamrock. “We should be in great fear of what this study is showing,” West says.
Widespread use of AI chatbots could reduce human trafficking in the scam industry, but it could also make the operations far harder to track and combat by removing the need for large-scale infrastructure like the compounds across Southeast Asia where human trafficking victims are housed and forced to work. “If one of their weak points is getting the people and having to maintain and feed and monitor these people, now they don’t have to do that,” says West. “Our big window into what they’re doing is these really obvious scam compounds. Now, they can do this in somebody’s two-bedroom apartment.”
This story originally appeared on wired.com.