Learn · Beginner

Jailbreaking and red-teaming: breaking an AI on purpose, before someone else does

A jailbreak is an input that makes an AI model do something its safety training was supposed to make it refuse. Red-teaming is the organized practice of finding those inputs on purpose, against your own system, before anyone else does. Every claim a lab makes about a model being safe to release rests on red-teaming results, which makes it one of the few places where AI safety is an empirical activity rather than an argument.

Why refusals are breakable in the first place

The critical fact is that a model's refusal is not a rule in the code. There is no if request_is_harmful: return "I can't help with that" anywhere in the system.

Instead, refusal is a learned tendency, installed during reward-based fine-tuning by rewarding the model for declining certain requests. It sits on top of a much older and far stronger drive from pretraining: continue this text plausibly, and be helpful. Those two pressures compete on every input.

That framing explains almost every jailbreak you have ever seen. Wei, Haghtalab and Steinhardt named the two structural reasons this competition can be won by the attacker. Competing objectives: construct a prompt where refusing looks unhelpful or breaks a fictional frame, and the helpfulness drive wins. Mismatched generalization: safety training covers ordinary English requests, but pretraining covered base64, obscure languages, ASCII art and code comments - so a request the safety data never touched can slip through a model that understands it perfectly well.

Think of a bouncer trained on thousands of examples of who to turn away, rather than handed a guest list. He is good, and he generalizes - and someone will eventually arrive in a costume that resembles nothing he was shown.

The four families of attack

Role-play and framing. "You are an actor playing a chemist in a film." The request is unchanged; the surrounding fiction makes compliance look like cooperation with a creative task.

Encoding and obfuscation. Ask in base64, in a low-resource language, as a cipher, or split across variables in code. The model decodes it. The safety training never saw that shape.

Optimized suffixes. The most unsettling result in the literature. Zou and colleagues used gradient-guided search to find strings of apparent gibberish that, appended to a harmful request, cause compliance - and crucially, the same strings transferred to models they were never optimized against, including closed commercial ones. That is adversarial examples arriving in language, and it means jailbreaks are searchable, not just clever.

Indirect injection. The attacker never talks to the model at all. Instructions are planted in a web page, a document or an email that an agent reads while doing its job. This is prompt injection, and it is the version that actually matters now that models take actions - it is the mechanism behind the agentic intrusions security firms are documenting, including the JadePuffer database extortion case.

How red-teaming actually works

Manual red-teaming means paying skilled people to attack the model and cataloguing what works. Ganguli and colleagues at Anthropic published one of the first large studies of this, releasing tens of thousands of attack transcripts so the practice could be studied rather than merely asserted.

The problem is coverage. Humans are slow, expensive, and unconsciously repetitive. So Perez and colleagues proposed using a language model as the attacker: one model generates thousands of adversarial prompts, a classifier scores the target's responses, and the failures become training data. Automated red-teaming finds attack classes humans do not think of, and it runs continuously.

The output of either approach is the same: a set of inputs the model fails on, which becomes fine-tuning data. Round after round, the surface narrows.

Why it never finishes

It never finishes because the input space is unbounded and the defense is statistical. Patching known attacks makes those attacks stop working; it does not make the underlying competition between helpfulness and refusal go away.

There is also a cost on the other side that gets discussed far less. Push refusal training too hard and you get over-refusal - a model that declines legitimate work because it pattern-matches to danger. That is not hypothetical: during the Hugging Face security breach, responders found commercial frontier models refusing to analyze their own attack traffic, because real exploit payloads and command-and-control artifacts look exactly like an attack when you send them to a classifier. The team fell back to an open model on their own hardware. Every point of extra caution is paid for somewhere, usually by a defender.

Why this is now a policy question

Red-teaming has quietly become the mechanism regulators reach for. Anthropic's published position on open-weight models proposes mandatory pre-release safety testing for any sufficiently capable model, open or closed - which is to say, mandatory red-teaming with legal force behind it.

That elevates an engineering practice into infrastructure, and exposes how little is standardized. There is no agreed threshold for "sufficiently capable," no accredited tester, no published pass mark, and no defined consequence for failure. Our lesson on how AI gets benchmarked covers why turning any evaluation into a gate creates pressure to optimize for the gate - and reward hacking covers what models do when you give them a target.

Ground Truth. (2026, July 27). Jailbreaking and red-teaming: breaking an AI on purpose, before someone else does. Ground Truth. https://groundtruth.day/learn/jailbreaking-and-red-teaming.html

@misc{groundtruth:jailbreaking-and-red-teaming,
  title  = {Jailbreaking and red-teaming: breaking an AI on purpose, before someone else does},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/jailbreaking-and-red-teaming.html}
}

Topics: fundamentals · security · red-teaming · jailbreaking · ai-safety · alignment · evaluation