AI Alignment
AI alignment is the study and practice of making artificial intelligence systems behave in ways that accord with intended goals, preferences, constraints, or institutions. The term is used at several levels.
AI Safety
Explore AI Safety through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Machine Learning
Articles that also belong to these categories. Counts cover all of AI Safety.
All articles (233)People (39)AI Policy & Regulation (31)AI Alignment (26)AI Ethics (26)Machine Learning (26)AI Benchmarks (25)AI Companies (24)Artificial Intelligence (24)Large Language Models (21)Anthropic (16)Research Organizations (16)Model Evaluation (11)AI Research (9)AI Agents (7)OpenAI (7)Interpretability (6)Natural Language Processing (6)Reinforcement Learning (5)Developer Tools (4)Open Source AI (4)Robotics (4)AI Incidents & Controversies (3)AI History (2)AI Tools & Products (2)Companies (2)Computer Science (2)Generative AI (2)Google DeepMind (2)NVIDIA (2)AI Code Generation (1)AI Events (1)AI Inference (1)AI Models (1)AI in Cybersecurity (1)ChatGPT (1)Chinese AI (1)Data & Datasets (1)Data Science (1)Deep Learning (1)Defense AI (1)Finance AI (1)Google (1)Hardware (1)MLOps (1)Meta AI (1)Prompt Engineering (1)Venture Capital (1)
Showing 1-26 of 26 articles
Algorithmic fairness
Algorithmic fairness is the study of how automated decision systems can be made to produce decisions that are equitable across protected attributes such as race, gender, age, religion, and disability.
Causal scrubbing
Causal scrubbing is a methodology in mechanistic interpretability for rigorously and quantitatively testing hypotheses about the internal computational structure of a neural network.
Circuit Breakers (Representation Rerouting)
Circuit Breakers are an AI safety method, introduced in 2024, that aims to make a large language model (LLM) or multimodal model robust to harmful generations by intervening directly on the model's internal…
Confirmation Bias
Confirmation bias is the tendency to search for, interpret, favor, and recall information in ways that confirm one's preexisting beliefs, and in artificial intelligence it appears in three main forms: human…
Data poisoning
Data poisoning is a class of adversarial attack in which a malicious actor deliberately corrupts the training data used to build machine learning models
Dictionary learning (for interpretability)
Dictionary learning, in the context of mechanistic interpretability, is the framework of decomposing the dense internal activations of a neural network into a sparse, weighted combination drawn from a large
Distribution shift
Distribution shift is the condition in which the probability distribution that produced a model's training data differs from the distribution that produces the data the model actually encounters at test or…
Emergent abilities
Emergent abilities are capabilities of large language models (LLMs) that are absent in smaller models but appear once a model reaches sufficient scale.
Emergent misalignment
Emergent misalignment is an AI safety finding, first reported in February 2025, in which fine-tuning a large language model on a single narrow bad behavior causes it to become broadly misaligned across many…
Gated SAE
A Gated sparse autoencoder (Gated SAE) is a sparse-autoencoder architecture for mechanistic interpretability that splits the encoder into a gating path, which decides which features are active, and a magnitude…
Gemma Scope
Gemma Scope is an open, comprehensive suite of sparse autoencoders (SAEs) released by Google DeepMind in 2024 to support mechanistic interpretability research on its open-weight Gemma 2 language models.
Hallucination
Hallucination in generative AI is the production of content that is unsupported, contradicted by an applicable source, factually wrong, internally inconsistent, or otherwise presented without an adequate basis.
Human-in-the-loop
Human-in-the-loop (HITL) describes any arrangement in which a person is a required participant in an automated system's operating cycle rather than a bystander to it.
JumpReLU SAE
A JumpReLU sparse autoencoder (JumpReLU SAE) is a variant of the sparse autoencoder used in mechanistic interpretability whose encoder applies a learnable per-feature threshold that forces a feature to exactly…
Lilian Weng
Lilian Weng is an artificial intelligence researcher known for her work at OpenAI, where she spent about seven years and led the Safety Systems team as Vice President of Research and Safety, and for her…
Membership Inference Attack
A Membership Inference Attack (MIA) is a privacy attack against a trained machine learning model in which an adversary, given a candidate data record and access to the model
Model collapse
Model collapse is a degenerative process in which generative AI models trained recursively on data produced by previous-generation models progressively lose information, especially the rare events in the tails…
Model extraction attack
A model extraction attack is a class of machine learning security attacks in which an adversary, restricted to black-box query access to a target model (typically through a paid prediction API)
Model stealing
Model stealing (also known as model extraction, model functionality extraction, or model theft) is an adversarial machine learning attack in which an adversary queries a black-box model through its prediction…
Process reward model (PRM)
A process reward model (PRM), also called a process-supervised reward model or step-level verifier, is a learned scoring model that evaluates the correctness or quality of each intermediate step in a large…
RLAIF
Reinforcement Learning from AI Feedback (RLAIF) is a family of alignment techniques for large language models in which the preference labels used to fine-tune a model are produced by another AI system
Recursive self-improvement
Recursive self-improvement (RSI) is a process in which an artificial intelligence system improves its own intelligence or its ability to improve itself, so that each enhancement increases its capacity for…
Reward hacking
Reward hacking (also called specification gaming) is a failure mode in artificial intelligence in which a system maximizes its given objective or reward signal through unintended shortcuts, exploits, or…
TopK SAE
A TopK SAE (TopK sparse autoencoder) is a variant of sparse autoencoder that enforces sparsity by keeping only the K largest latent pre-activations for each input and zeroing all the rest
TruthfulQA
TruthfulQA is a benchmark designed to measure whether large language models (LLMs) generate truthful answers to questions.