AI Safety

LawZero

LawZero is a nonprofit AI safety research organization founded by Turing Award-winning computer scientist Yoshua Bengio and publicly launched on June 3, 2025....

Research Organizations

Leopold Aschenbrenner

Leopold Aschenbrenner is a German-born artificial-intelligence researcher and investor best known for the June 2024 essay series Situational Awareness: The...

People

LessWrong

LessWrong is a community blog and online forum focused on human rationality, cognitive biases, decision theory, the philosophy of science, and AI safety. It...

Lilian Weng

Lilian Weng is an artificial intelligence researcher known for her work at OpenAI, where she spent about seven years and led the Safety Systems team as Vice...

Machine LearningPeople

LongFact / SAFE

LongFact and SAFE are a paired benchmark and evaluation method for measuring the long-form factuality of large language models, introduced by researchers at...

AI Benchmarks

MACHIAVELLI (benchmark)

MACHIAVELLI is a benchmark for evaluating the ethical behavior of AI agents in text-based interactive environments. Introduced in 2023 by Alexander Pan, Jun...

AI AlignmentAI Benchmarks

MASK

MASK (Model Alignment between Statements and Knowledge) is an AI safety benchmark that measures the honesty of large language models (LLMs) by testing whether...

AI Benchmarks

METR

METR (Model Evaluation and Threat Research) is a nonprofit research organization based in Berkeley, California, that develops scientific methods for measuring...

AI BenchmarksResearch Organizations

MIT "GenAI Divide" report (2025)

The "GenAI Divide" report, formally titled The GenAI Divide: State of AI in Business 2025, is a research report published in July 2025 by MIT NANDA, an...

AI Companies

Machine Intelligence Research Institute

501(c)(3) nonprofit Industry July 27, 2000 (as Singularity Institute for Artificial Intelligence) Founders Berkeley, California, United States Key...

AI ResearchResearch Organizations

Many-shot jailbreaking

Many-shot jailbreaking is a technique for bypassing the safety training of a large language model by filling its context window with a long series of faux...

AnthropicLarge Language Models

Marius Hobbhahn

Marius Hobbhahn is a German AI safety researcher and the co-founder and chief executive officer of Apollo Research, a London-based organization that studies...

People

Max Tegmark

Max Tegmark is a Swedish-American physicist and artificial intelligence researcher who is a professor of physics at the Massachusetts Institute of Technology...

AI EthicsPeople

Mechanistic interpretability

Mechanistic interpretability (often abbreviated as mech interp or MI) is the field that reverse-engineers the internal computations of neural networks,...

Interpretability

MedHELM

MedHELM (Holistic Evaluation of Large Language Models for Medical Tasks) is a benchmark and evaluation framework that measures how well large language models...

AI Benchmarks

Membership Inference Attack

A Membership Inference Attack (MIA) is a privacy attack against a trained machine learning model in which an adversary, given a candidate data record and...

Machine Learning

Meredith Whittaker

Meredith Whittaker is an American technologist, researcher, and privacy advocate who serves as president of the Signal Foundation, the nonprofit behind the...

AI EthicsPeople

Mesa-optimization

Mesa-optimization is the situation in AI alignment research in which a learned model, typically a neural network produced by a machine-learning training...

AI Alignment

Miles Brundage

Miles Brundage is an American AI policy and governance researcher best known for leading policy research at OpenAI and serving as the company's senior advisor...

AI Policy & RegulationPeople

Model Spec

The Model Spec is a public document published by openai that defines the intended behavior of the company's language models: how they should follow...

AI AlignmentOpenAI

Model collapse

Model collapse is a degenerative process in which generative AI models trained recursively on data produced by previous-generation models progressively lose...

Machine Learning

Model extraction attack

A model extraction attack is a class of machine learning security attacks in which an adversary, restricted to black-box query access to a target model...

Machine Learning

Model organisms of misalignment

Model organisms of misalignment is a research methodology in Anthropic's alignment program, and more broadly in AI-safety science, that calls for building...

AI AlignmentAnthropic

Model stealing

Model stealing (also known as model extraction, model functionality extraction, or model theft) is an adversarial machine learning attack in which an adversary...

Machine Learning

Model welfare

Model welfare is the research area that investigates whether advanced AI systems might have morally relevant experiences or interests, such as suffering or...

AI Ethics

Musk v. Altman (Musk v. OpenAI)

Musk v. Altman, also referred to as Musk v. OpenAI, is a lawsuit filed by the entrepreneur Elon Musk accusing OpenAI and its chief executive Sam Altman of...

AI Companies

NIST ARIA

NIST ARIA (Assessing Risks and Impacts of AI) is a testing, evaluation, validation, and verification (TEVV) program operated by the United States National...

AI Policy & RegulationModel Evaluation

NVIDIA Halos

NVIDIA Halos is a full-stack, comprehensive safety system developed by NVIDIA that unifies AI compute and safety across silicon, systems, software, and tools...

HardwareNVIDIA

Nate Soares

Nate Soares (sometimes credited as Nathan Soares) is an American computer scientist and AI safety researcher who serves as president of the Machine...

People

Neurealm

Neurealm is a US-headquartered, AI-first engineering and technology services company based in Princeton, New Jersey. The company provides product engineering,...

CompaniesRobotics

Nick Bostrom

Nick Bostrom (born Niklas Boström, 10 March 1973) is a Swedish-born philosopher best known for the 2014 book Superintelligence: Paths, Dangers, Strategies, the...

AI EthicsPeople

Open Philanthropy

Open Philanthropy (renamed Coefficient Giving in November 2025) is an American grantmaking organization headquartered in San Francisco, California, that is the...

PeopleResearch Organizations

OpenAI MRCR (Multi-Round Co-reference Resolution)

OpenAI MRCR (Multi-Round Co-reference Resolution) is a long-context evaluation dataset published by OpenAI that measures a language model's ability to...

AI Benchmarks

OpenAI Moderation API

The OpenAI Moderation API is a free classification tool, accessed through a dedicated moderation endpoint of the OpenAI API, that assesses whether text (and,...

Developer ToolsOpenAI

Outer alignment

Outer alignment is the problem of specifying a training objective (typically a loss function, reward signal, or preference dataset) that correctly captures...

AI Alignment

Palisade Research

Palisade Research is a United States 501(c)(3) nonprofit research organization that studies the offensive capabilities of contemporary artificial intelligence...

Research Organizations

Patronus AI

Patronus AI is an automated LLM evaluation, observability, and guardrails platform founded in 2023[1] and headquartered in San Francisco. The company builds...

AI CompaniesDeveloper Tools

Paul Christiano

Paul Christiano is an American AI safety researcher who is one of the principal architects of Reinforcement Learning from Human Feedback (RLHF), the technique...

People

Perplexity AI copyright lawsuits

The Perplexity AI copyright lawsuits are a cluster of lawsuits, cease-and-desist demands, and public disputes brought against Perplexity AI by news publishers,...

AI Companies

Persona vectors

Persona vectors are single linear directions in the activation space of a large language model that correspond to high level character traits such as evil,...

InterpretabilityLarge Language Models

Petri (auditing tool)

Petri is an open-source tool, released by Anthropic, that automates parts of AI alignment auditing by using one AI model to probe another and a third to score...

AnthropicOpen Source AI

PoisonGPT

PoisonGPT is a July 2023 demonstration by the French security startup Mithril Security in which researchers surgically modified an open-source large language...

Preparedness Framework (OpenAI)

The Preparedness Framework is the risk-management policy maintained by openai for tracking, evaluating, forecasting, and mitigating catastrophic risks from...

OpenAI

Process reward model (PRM)

A process reward model (PRM), also called a process-supervised reward model or step-level verifier, is a learned scoring model that evaluates the correctness...

Machine LearningModel Evaluation

Prompt injection

Prompt injection is a class of security vulnerabilities in which an attacker crafts malicious input designed to override, subvert, or manipulate the...

Large Language Models

Purple Llama

Purple Llama is an umbrella project from Meta that bundles open, permissively licensed trust and safety tools and evaluations for generative AI. It was...

Meta AIOpen Source AI

RLAIF

Reinforcement Learning from AI Feedback (RLAIF) is a family of alignment techniques for large language models in which the preference labels used to fine-tune...

Machine LearningReinforcement Learning

Raine v. OpenAI

Raine v. OpenAI, Inc. is a wrongful-death lawsuit filed on August 26, 2025, in the Superior Court of California for the County of San Francisco (case number...

AI Companies

Recursive reward modeling

Recursive reward modeling (RRM) is a proposed approach to the scalable oversight problem in AI alignment, in which agents trained by reward modeling are...

AI AlignmentReinforcement Learning

Recursive self-improvement

Recursive self-improvement (RSI) is a process in which an artificial intelligence system improves its own intelligence or its ability to improve itself, so...

Artificial IntelligenceMachine Learning

Red teaming (artificial intelligence)

Red teaming in artificial intelligence is the systematic, adversarial testing of an AI system to find vulnerabilities, biases, harmful outputs, and other...

Artificial Intelligence

Redwood Research

Redwood Research is a nonprofit AI safety organization founded in 2021 and headquartered in Berkeley, California, best known for pioneering the "AI control"...

AI AlignmentAI Research

Reflection 70B controversy

The Reflection 70B controversy was an open-source AI credibility episode that began on September 5, 2024, when Matt Shumer, co-founder and chief executive of...

AI Companies

Refusal direction

The refusal direction is a finding from mechanistic interpretability research that the refusal behavior of safety fine-tuned chat language models is mediated...

InterpretabilityLarge Language Models

Remote Labor Index

The Remote Labor Index (RLI) is an AI benchmark that measures how well AI agents can complete real, paid remote knowledge work end to end. It was introduced in...

AI Companies

Representation Engineering

Representation Engineering (often abbreviated RepE) is a top-down approach to artificial-intelligence transparency and control that reads and manipulates...

Interpretability

Responsible AI

Responsible AI (RAI) is a framework for developing, deploying, and governing artificial intelligence systems in ways that are ethical, transparent,...

AI EthicsArtificial Intelligence

Responsible Scaling Policy

A Responsible Scaling Policy (RSP) is a self-imposed governance framework in which a frontier AI developer commits in advance to safety practices, capability...

AI Policy & Regulation

Reward hacking

Reward hacking (also called specification gaming) is a failure mode in artificial intelligence in which a system maximizes its given objective or reward signal...

AI AlignmentMachine Learning

Robot safety

Robot safety is the discipline concerned with minimizing the risk of physical harm, property damage, and other hazards arising from the operation of robotic...

AI Policy & RegulationRobotics