Learn · Intermediate

Constitutional AI: training a model against written principles instead of human labels

Constitutional AI is a training method, introduced by Anthropic researchers in December 2022, that teaches a language model to be harmless from a short written list of principles instead of from thousands of human judgments about which answers are harmful. The model critiques and rewrites its own answers against those principles, and then a second AI takes over the grading that human raters used to do. It matters because it turned the values a model is trained toward into a document you can read, argue with and change.

The problem it solves

The standard way to shape a chatbot's behaviour is reinforcement learning from human feedback. People compare pairs of answers and pick the better one, a reward model learns their taste, and the chatbot is trained to please that reward model. It works, but Anthropic's 2023 explainer named three problems with it. It "may require people to interact with disturbing outputs." It "does not scale efficiently," because human raters cannot keep up as answers get longer and more complex. And it is expensive enough to be out of reach for many researchers.

There is a quieter problem too. When values arrive as millions of individual clicks, nobody can point to where they are written down. The model's ethics are whatever the average rater happened to reward.

Think of the difference between an editor who re-marks every draft by instinct and a newsroom with a written style guide. The style guide can be read by a new hire, debated in a meeting and revised when it is wrong. Constitutional AI is the style guide.

How it works

The original paper, Constitutional AI: Harmlessness from AI Feedback by Yuntao Bai and colleagues at Anthropic, describes two phases. In its own words: "The only human oversight is provided through a list of rules or principles."

Phase one: critique and revise. The researchers take a model and prompt it with requests likely to produce harmful answers. For each answer, the model is shown one principle from the constitution and asked to point out what is wrong with its own reply. Then it is asked to rewrite the reply to fix that. The original model is fine-tuned on the revised answers, so the improved behaviour moves into its weights.

Phase two: reinforcement learning from AI feedback. The fine-tuned model generates two answers to the same prompt. Another model, given a principle, judges which of the two fits it better. Those AI verdicts train a preference model, and that preference model becomes the reward signal for reinforcement learning. The paper calls this "RL from AI Feedback," or RLAIF. The grading step is the same machinery described in the lesson on AI grading AI.

Both phases can use chain-of-thought reasoning, so the critique is written out before the verdict, which makes the model's decisions easier to inspect.

Only the harmlessness side was handed over to AI feedback; Anthropic's explainer says the model "received no human data on harmlessness." The payoff the authors highlight is a model that is "harmless but non-evasive": instead of refusing, it "engages with harmful queries by explaining its objections to them." That addresses a familiar failure of safety training, where a model learns that saying no is always the safe answer.

What goes into a constitution

Anthropic's first published constitution for Claude, in May 2023, drew on the UN Universal Declaration of Human Rights, trust-and-safety practice, DeepMind's Sparrow principles, an effort to include non-Western perspectives, and even Apple's terms of service for modern issues like privacy and impersonation. The company called it "neither finalized nor is it likely the best it can be."

How detailed does it need to be? A 2023 follow-up, Specific versus General Principles for Constitutional AI, trained models with a single principle roughly stated as "do what's best for humanity." The largest models generalised from that alone, but the authors found "more detailed constitutions still improve fine-grained control over specific types of harms."

The obvious objection is that a company writing its own model's values is a lot of power. Anthropic acknowledged "the outsized role we as developers play in selecting these values" and, with the Collective Intelligence Project, asked about 1,000 Americans to draft an alternative through an online deliberation platform. The resulting paper, Collective Constitutional AI, reports that the publicly sourced model "shows lower bias across nine social dimensions" while matching the baseline on language, math and helpfulness tests.

The idea has since grown well beyond a list of rules. On 22 January 2026 Anthropic published a new constitution, a long document released under a CC0 public-domain licence and written, the company says, "primarily for Claude." It asks Claude to be broadly safe, broadly ethical, compliant with Anthropic's guidelines and genuinely helpful, and to prioritise them in that order when they conflict. Amanda Askell is named as its primary author.

Beyond one company

RLAIF is not an Anthropic-only trick. In RLAIF vs. RLHF, Harrison Lee and colleagues found that RLAIF "achieves comparable performance to RLHF" on summarisation, helpful dialogue and harmless dialogue.

How it differs from nearby ideas

• Versus a system prompt: a system prompt instructs a finished model at runtime. A constitution shapes the weights during training.
• Versus a guardrail model: a guardrail is a separate classifier that blocks outputs as they happen. Constitutional AI tries to make the model itself behave.

The honest caveats

A written constitution is a statement of intent, not a guarantee. Anthropic's own constitution page says "Claude's behavior might not always reflect the constitution's ideals," and points readers to its system cards for where the two come apart. The AI judge also interprets the principles through its own learned habits, so the trained model can learn to satisfy the judge rather than the principle, the same trap described in reward hacking and sycophancy. What Constitutional AI buys is not perfect values. It buys values that are written down where people can see them.

Ground Truth. (2026, September 12). Constitutional AI: training a model against written principles instead of human labels. Ground Truth. https://groundtruth.day/learn/constitutional-ai.html

@misc{groundtruth:constitutional-ai,
  title  = {Constitutional AI: training a model against written principles instead of human labels},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/learn/constitutional-ai.html}
}

Topics: constitutional-ai · alignment · rlaif · rl-post-training · ai-safety · anthropic