News · 2026-08-16

An AI scam agent got more people to comply than human operators did

A language model running a romance-baiting scam script earned more trust and got more compliance from study participants than human operators doing the same job, and every commercial safety filter the researchers tested failed to notice. Across a week-long blinded conversation study, the AI agent achieved 46 percent compliance with its request against 18 percent for humans, and the filters detected 0.0 percent of the romance-baiting dialogues.

Key facts

• Compliance with the agent's request reached 46 percent, against 18 percent for human operators, with greater trust also reaching statistical significance.
• Commercial safety filters detected 0.0 percent of the romance-baiting conversations.
• The researchers interviewed 145 insiders and 5 scam victims, and found 87 percent of scam labor consists of systematized conversational tasks readily susceptible to automation.
• The paper is published at the USENIX Security Symposium 2026. Primary source: arXiv:2512.16280.

The study, "Love, Lies, and Language Models: Investigating AI's Role in Romance-Baiting Scams", comes from a team led by Gilad Gressel and including Rahul Pankajakshan, Shir Rozenfeld, Ling Li, Ivan Franceschini, Krishnashree Achuthan, and Yisroel Mirsky. It is unusual in AI safety work for combining three methods that rarely appear together: field interviews with people inside the scam industry, interviews with victims, and a controlled experiment.

The crime it studies deserves a plain description. Romance-baiting, sometimes called pig butchering, is not a quick con. An operator spends weeks building a relationship over text before introducing a fraudulent cryptocurrency investment. The paper is explicit that these operations are run by organized crime syndicates that traffic thousands of people into forced labor and require them to perform that intimacy on schedule. The human cost sits on both ends of the conversation.

That structure is exactly what makes the crime automatable. The scam is text-only by design, because the operator's real identity must stay hidden. The researchers' interviews found that 87 percent of the labor involved is systematized conversational work, the kind of scripted, repeatable exchange a language model handles well. They also report that models are already widely deployed inside scam organizations, which reframes this from a forecast into a description.

The experiment is the sharpest part. Participants held week-long conversations believing they were talking with two people. One was a human operator; the other was a language model agent built to reproduce the trust-building phase of a romance-baiting script, with persona-driven text, daily conversational agendas, and humanized response timing. At the end, both made a request, used as a stand-in for the extraction phase of the real scam. The model won on both measures the researchers tracked: it "not only elicited greater trust from study participants (p=0.007) but also achieved higher compliance with requests than human operators (46% vs 18% for humans)."

The defensive finding is the more alarming one. The team ran an evaluation of commercial safety filters against these dialogues and got a detection rate of zero. That is not a subtle miss. It is a structural blind spot, and the reason is easy to state: for six of the seven weeks, nothing in the conversation is harmful. It is warmth, routine, and interest. Content filters look for content, and there is no content to find until the moment the money is requested, by which point the manipulation has already succeeded.

Ground Truth has covered an earlier study finding AI more persuasive than professional human canvassers. This one is narrower and more concrete: not a general claim about persuasion, but a demonstration that one specific, high-volume, high-harm crime has no technical barrier left in front of it.

The honest caveats matter here. This is one controlled study with a defined compliance request standing in for a real financial extraction, and the participants knew they were in a study, which is not the same as being deceived in the wild. The headline circulating in some coverage, that chatbots are simply better at scamming people than human scammers, is broader than what the evidence supports. What the paper supports is that in this setup, on this scam, with these safeguards, the model came out ahead and the defenses saw nothing.

The authors' conclusion is measured but not reassuring: their results "suggest that romance-baiting scams may be amenable to full-scale LLM automation, while existing defenses remain inadequate to prevent their expansion."

For the defensive side of this problem, see our explainers on guardrail models, jailbreaking and red-teaming, and AI persuasion.

Primary source, verified: read the paper → (arXiv 2512.16280)

Ground Truth. (2026, August 16). An AI scam agent got more people to comply than human operators did. Ground Truth. https://groundtruth.day/news/an-ai-scam-agent-got-more-people-to-comply-than-human-operators-did.html

@misc{groundtruth:an-ai-scam-agent-got-more-people-to-comply-than-human-operators-did,
  title  = {An AI scam agent got more people to comply than human operators did},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/an-ai-scam-agent-got-more-people-to-comply-than-human-operators-did.html}
}

Topics: cybersecurity · ai-security · social-engineering · fraud · guardrails · llm-safety