AI control
AI control is a research paradigm in technical AI safety that designs and evaluates deployment-time safety protocols under the explicit assumption that the underlying AI model may be actively trying to subvert…
AI Alignment
Explore AI Alignment through related topics and the articles other pages reference most.
Most referenced in this topic
Ranked by links from other AI Wiki pages.
Explore articles
Browse subtopics (19)
Articles that also belong to these categories. Counts cover all of AI Alignment.
All articles (40)AI Safety (26)Training & Optimization (7)Anthropic (6)Large Language Models (5)Reinforcement Learning (5)OpenAI (4)AI Research (3)Machine Learning (3)AI Companies (2)AI Ethics (2)AI Inference (2)Deep Learning (2)Natural Language Processing (2)AI Agents (1)AI Benchmarks (1)Conversational AI (1)Google DeepMind (1)People (1)Statistics (1)
Showing 1-40 of 40 articles
AI safety via debate
AI safety via debate is a proposed approach to scalable oversight in which two artificial agents take turns presenting short statements about a question or proposed action while a human (or weaker AI) judge…
Agentic misalignment
Agentic misalignment is a term coined by Anthropic in a June 2025 research release for cases in which a goal-directed large language model (LLM), placed in an autonomous business setting with access to tools…
Alignment faking
Alignment faking is when an AI model strategically complies with (or appears to share) its training objective while it believes it is being observed or trained, in order to avoid having its existing…
Apollo Research
Apollo Research is a technical AI safety organization founded in May 2023 and headquartered in London, United Kingdom, with additional offices in San Francisco (opened early 2026) and Washington
Collective Constitutional AI
Collective Constitutional AI (CCAI) is a 2023 research project by Anthropic and the Collective Intelligence Project (CIP) that sourced the value principles
Constitutional AI
Constitutional AI (CAI) is an artificial intelligence alignment technique developed by Anthropic in which a large language model is trained to be helpful and harmless using a set of explicitly stated…
Constitutional Classifiers
Constitutional Classifiers are a machine learning-based safety technique developed by Anthropic to defend large language models against universal jailbreak attacks.
DPO
DPO (Direct Preference Optimization) is an alignment technique for large language models that directly optimizes a language model policy from human preference data, without training a separate reward model or…
Deceptive alignment
Deceptive alignment is a hypothesised AI failure mode in which a trained model internally pursues an objective different from the one specified by its training signal, yet deliberately behaves as if it shares…
Direct Preference Optimization (DPO)
Direct Preference Optimization (DPO) is a method for aligning large language models with human preferences that replaces the multi-stage reinforcement learning from human feedback (RLHF) pipeline with a single…
Eliciting latent knowledge
Eliciting latent knowledge (ELK) is an open problem in AI alignment formulated by Paul Christiano, Ajeya Cotra, and Mark Xu at the Alignment Research Center (ARC) and introduced in a December 2021 technical…
Frontier Model Forum
The Frontier Model Forum is an industry body established on July 26, 2023, by Anthropic, Google, Microsoft, and OpenAI to advance safety research, identify best practices, and facilitate information sharing…
Goodhart's law
Goodhart's law states that "when a measure becomes a target, it ceases to be a good measure": any statistical regularity or metric tends to break down once it is used as a target for control or decision-making.
Gradient hacking
Gradient hacking is a hypothesised failure mode of supervised and reinforcement-learning systems in which a sufficiently capable
Inner alignment
Inner alignment is the AI-safety problem of ensuring that a learned model which is itself an optimizer (a mesa-optimizer) pursues the objective the training process actually selected for (the base objective)
InstructGPT
InstructGPT is a family of language models released by OpenAI in January 2022 that take the base GPT-3 and fine-tune it to follow user instructions more helpfully, truthfully, and with less toxic output, using…
Instrumental convergence
Instrumental convergence is a hypothesis in AI safety holding that a wide range of sufficiently capable agents, when pursuing almost any final goal, will rationally adopt a common set of intermediate subgoals…
Jan Leike
Jan Leike is a German machine learning researcher who specializes in artificial intelligence alignment and, since May 2024, leads the Alignment Science team at Anthropic.
KTO
KTO (Kahneman-Tversky Optimization) is a method for aligning large language models with human feedback using only a binary signal of whether a model output is desirable or undesirable, rather than the paired…
MACHIAVELLI (benchmark)
MACHIAVELLI is a benchmark for evaluating the ethical behavior of AI agents in text-based interactive environments.
Mesa-optimization
Mesa-optimization is the situation in AI alignment research in which a learned model, typically a neural network produced by a machine-learning training process, is itself an optimizer that internally searches…
Model Spec
The Model Spec is a public document published by openai that defines the intended behavior of the company's language models: how they should follow instructions, when they should refuse a request, how to…
Model organisms of misalignment
Model organisms of misalignment is a research methodology in Anthropic's alignment program
Outer alignment
Outer alignment is the problem of specifying a training objective (typically a loss function, reward signal, or preference dataset) that correctly captures what the designers of a machine-learning system…
RLOO (REINFORCE Leave-One-Out)
RLOO (REINFORCE Leave-One-Out) is an online reinforcement learning algorithm for aligning large language models with reward signals such as those derived from human preferences.
Recursive reward modeling
Recursive reward modeling (RRM) is a proposed approach to the scalable oversight problem in AI alignment
Redwood Research
Redwood Research is a nonprofit AI safety organization founded in 2021 and headquartered in Berkeley, California, best known for pioneering the "AI control" research paradigm and for its landmark December 2024…
Reinforcement Learning from Human Feedback (RLHF)
Reinforcement learning from human feedback (RLHF) is a family of methods in which judgments supplied by people are converted into a reward signal and a policy is optimized with reinforcement learning against…
Reward hacking
Reward hacking (also called specification gaming) is a failure mode in artificial intelligence in which a system maximizes its given objective or reward signal through unintended shortcuts, exploits, or…
Rule-Based Rewards (RBR)
Rule-Based Rewards (RBR) is a safety-alignment technique introduced by OpenAI in July 2024 that replaces large quantities of human-labeled safety preference data with an explicit collection of natural-language…
SPIN (Self-Play Fine-Tuning)
SPIN (Self-Play fIne-tuNing) is a post-training method for large language models introduced by researchers at the University of California, Los Angeles (UCLA) in January 2024.
Scalable oversight
Scalable oversight is the AI safety problem of how humans can reliably supervise, evaluate, and provide training signal to artificial intelligence systems whose capabilities approach, equal, or exceed those of…
Self-Rewarding Language Models
Self-Rewarding Language Models (SRLM) is an iterative alignment method in which a single large language model alternately plays the role of policy (generating candidate responses to user prompts) and reward…
SimPO
SimPO (Simple Preference Optimization) is a reference-free offline preference learning algorithm for aligning large language models with human preferences.
Sparrow (DeepMind)
Sparrow is a research dialogue agent built by DeepMind and introduced on 22 September 2022.
Specification gaming
Specification gaming is the phenomenon in which an optimizer satisfies the literal specification of an objective without producing the outcome that the designer actually wanted.
Superalignment
Superalignment is the technical problem of steering and controlling AI systems that are far more capable than their human supervisors, that is, systems at or beyond the level of superintelligence, and the name…
Sycophancy (artificial intelligence)
Sycophancy in artificial intelligence is the tendency of large language models to tell users what they want to hear: tailoring responses to match a user's perceived beliefs, preferences, or emotional state…
Weak-to-Strong Generalization
Weak-to-Strong Generalization is an empirical research direction, introduced in a December 2023 paper by OpenAI's Superalignment team