πŸ“œ

Constitutional AI (CAI)

A method to improve AI itself using a constitution (principles)

Constitutional AI (CAI) is a technique proposed by Anthropic in 2022 and is the core alignment technology of Claude. In RLHF, having humans directly evaluate responses is expensive and difficult to scale. CAI provides AI with a Constitution β€” principles such as "be helpful without being harmful" and "avoid discriminatory expressions." In the Critique phase, AI critiques its own responses against the principles, and in the Revision phase, it revises responses reflecting the critique. This self-improvement data is used for SFT and RLAIF (Reinforcement Learning from AI Feedback).

Key Concepts

1

Define Constitution β€” write 10-20 principles ("must be harmless", "must be accurate", etc.)

2

Red Teaming: collect initial model responses with harmful prompts

3

Critique: AI critiques its own responses against constitutional principles ("this response violates principle X")

4

Revision: AI revises responses reflecting the critique ("to comply with principle X...")

5

Perform SFT with revised responses

6

RLAIF: train reward model with AI-generated preference data β†’ reinforcement learning (AI evaluation instead of human evaluation)

Pros

  • ✓ Significantly reduced human evaluation costs (RLAIF)
  • ✓ Principles explicitly documented β†’ auditable
  • ✓ Scalable alignment
  • ✓ Behavior adjustable by modifying principles

Cons

  • ✗ Depends on AI self-critique capability
  • ✗ Possible conflicts between constitutional principles
  • ✗ Blind spots exist if principles are incomplete
  • ✗ Difficult to fully replace human evaluation

Use Cases

Claude (all Anthropic models) AI safety research Multilingual alignment (applying cultural principles) Custom enterprise AI value settings

References

Related

πŸ‘₯
RLHF
Reinforcement Learning from Human Feedback
→
← 🎯 AI Alignment πŸŒ€ Hallucination →