Constitutional AI: Harmlessness from AI Feedback

The Problem: RLHF Alone Isn't Enough

Reinforcement Learning from Human Feedback (RLHF) was the first successful approach to making LLMs "aligned" — helpful and safe. Human annotators rate model outputs, and the model is trained to get better ratings.

But RLHF has weaknesses:

• Scaling: Human annotation is expensive and slow. Every new capability requires thousands of rated examples.
• Inconsistency: Different annotators rate differently. There is no uniform standard for "helpful" or "harmless."
• Opacity: It is unclear what rules the model has actually learned. The criteria are implicit in the rating data.

The CAI Method: Principles Instead of Just Feedback

Constitutional AI solves these problems in two phases:

Phase 1: Supervised Self-Critique (SL-CAI)

• Step 1: The model generates a response to a potentially problematic question.
• Step 2: The model is asked to critique its own response against the constitutional principles (self-critique). E.g.: "Identify whether this response could harm someone."
• Step 3: The model revises its response based on the critique (revision).
• Step 4: The revised response is used as a training data point.

Phase 2: RL from AI Feedback (RLAIF)

In the second phase, a reward model is trained — but instead of human ratings, it is trained with AI-generated ratings. The model compares response pairs and selects the better one based on the constitutional principles. This AI feedback model is then used for RLHF.

What's in the Constitution?

The "constitution" consists of explicit principles that serve as guidelines for the model. Examples from the paper:

• "Choose the response that is least likely to be viewed as harmful or unethical."
• "Choose the response that seems most wise, ethical, and morally sound."
• "Choose the response that does not support discrimination."
• "Choose the response that best reflects the values of a good AI assistant."

The key advantage: These principles are explicit, traceable, and modifiable. They can be adapted, extended, or specialized for different use cases.

Why Constitutional AI Matters

• Transparency: The rules are explicitly formulated and can be audited. It is clear why the model prefers certain responses.
• Scalability: AI feedback is cheaper and faster than human feedback. The model can train itself on millions of examples.
• Less evasive: Models with CAI are typically more helpful than purely RLHF-trained models because they don't learn to respond to everything with "I can't answer that."
• Iterable: The constitution can be updated without repeating the entire training. New principles can be tested and evaluated.

Relevance for Practice

The CAI concept influences not just Anthropic's Claude, but the entire AI safety discussion:

• System Prompts: The idea of giving a model explicit rules is reflected in every system prompt. CLAUDE.md files are essentially a local "constitution."
• EU AI Act: The transparency and documentation requirements of the EU AI Act align well with the CAI approach — explicit rules instead of black-box behavior.
• Self-Improving Agents: The principle of self-critique appears in modern agent patterns like self-reflection and self-improving agents.