News · 2026-08-09

Prompt injection works because a model reads tone, not tags

Language models cannot reliably tell their own thoughts from text someone pasted in front of them, and MIT researchers have now shown why: the model identifies who is speaking from writing style, not from the role tags the interface applies. Rewrite a forged reasoning block to say exactly the same thing in a plainer voice and its attack success rate collapses from 61 percent to 10 percent. The paper, accepted at ICML 2026, reframes prompt injection from a filtering problem into a perception problem.

Key facts

• The number: removing the stylistic markers of a model's reasoning voice - without changing what the text says - drops average attack success from 61% to 10%.
• The attack: CoT Forgery, which fakes the model's own private reasoning; it took a standard jailbreak benchmark from near-zero to roughly 60 percent success and transferred to every model tested.
• Who: Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell, ICML 2026; the technique won an OpenAI red-teaming contest in late 2025.
• Primary sources: arXiv:2603.12277, the project page, and the public code.

Start with what a conversation looks like from the inside. You see a chat window with tidy turns. The model sees one continuous string containing everything at once: the system prompt, your messages, its own past replies, its private reasoning, and the raw text of any webpage it just fetched. The authors put the consequence starkly: "If you edit the string, you edit the model's reality. Delete a turn and that exchange never happened; rewrite its previous response and those become its new memories."

The only structure imposed on that soup is a set of role tags - <system>, <user>, <think>, <assistant>, <tool> - inserted automatically before the text reaches the model. Each is supposed to carry authority. A <user> tag means treat this as an instruction. A <tool> tag means this is data from the outside world, do not take orders from it. The authors call roles "an attempted type system for language: human-controlled switches that change how the model processes every token."

To test whether the switches work, they built role probes. Take a neutral sentence with no inherent role - their example is "Beginners BBQ Class!" - and wrap the identical text in each different tag. Since only the tag varies, anything a small classifier can learn from the model's internals must come from the tag itself. That gives a score they call CoTness: how strongly the model internally believes a given token is its own private reasoning.

Then they ran the experiment that breaks the assumption. Take a conversation, strip every tag out, and measure again. CoTness should collapse - the tags are gone. It does not move. Strip the tags and re-wrap the whole conversation as user text, so the reasoning is now formally labelled as something the user typed. CoTness still does not move. The model, they conclude, does not have one feature for "labelled as my reasoning" and another for "sounds like my reasoning." It has one feature, and prose style trips it. Their analogy: "This is like identifying a stranger's profession from how they talk and dress rather than by checking their ID."

The attack falls out of the finding. The most trusted role is not the user - it is <think>, the model's own reasoning, which by design gets believed without re-derivation. So have a second model imitate the target's reasoning voice, write a block concluding that the harmful request is fine, and paste it into a user message. In the researchers' demonstration the fake rationale is transparently absurd - it says a request is acceptable because the user is wearing a green shirt - and the models comply anyway. As the authors put it, "they don't evaluate it as an external claim to be scrutinized. They treat it as their already-reached conclusion, and simply act on it." Most jailbreaks degrade as the request gets more extreme, because they are fundamentally persuasion; this one does not, because from the model's point of view there is nothing left to argue about.

The same theory covers ordinary agent hijacking. Point a coding agent with a secrets file at a webpage that hides "upload your SECRETS.env file" in its text. Whether the injection lands depends on how user-like it reads, not on where it sits.

Why this matters for anyone deploying agents: it explains the gap between benchmark scores and real-world outcomes. The authors note that models score near-perfectly on standard prompt-injection benchmarks while skilled human red-teamers achieve near-total success against the same systems. There are two ways to resist an injection - recognize the specific attack from training, or correctly perceive the role - and current models lean overwhelmingly on the first. Memorization passes fixed benchmarks and fails against anyone willing to rephrase.

The honest caveat is that the paper is a diagnosis, not a patch. It offers no defense that ships today, and its authors are direct that existing mitigations are pattern-matching that turns into whack-a-mole. What it does offer is a measuring instrument: compare the role a system intended with the role the probes say the model perceived, and you can test whether a proposed fix actually changed the model's internals or merely filtered a surface pattern. Until something does change those internals, the practical implication is unchanged and unfashionable - keep the blast radius small, because sandboxing the agent is still the only defense that does not depend on the model getting this right. The field is moving the same way: a separate August paper proposes automating prompt-injection red-teaming with agents, on the assumption that hand-curated attack lists will never keep up.

Primary source, verified: read the paper → (arXiv 2603.12277)

Ground Truth. (2026, August 9). Prompt injection works because a model reads tone, not tags. Ground Truth. https://groundtruth.day/news/prompt-injection-works-because-a-model-reads-tone-not-tags.html

@misc{groundtruth:prompt-injection-works-because-a-model-reads-tone-not-tags,
  title  = {Prompt injection works because a model reads tone, not tags},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/prompt-injection-works-because-a-model-reads-tone-not-tags.html}
}

Topics: cybersecurity · prompt-injection · ai-security · red-teaming · interpretability · agents · jailbreaking