AI Safety

Explore AI Safety through related topics and the articles other pages reference most.

Explore articles

Browse subtopics (55)

Articles that also belong to these categories. Counts cover all of AI Safety.

Showing 181-240 of 278 articles

Model extraction attack

A model extraction attack is a class of machine learning security attacks in which an adversary, restricted to black-box query access to a target model (typically through a paid prediction API)

Machine Learning

Model stealing

Model stealing (also known as model extraction, model functionality extraction, or model theft) is an adversarial machine learning attack in which an adversary queries a black-box model through its prediction…

Machine Learning

Model welfare

Model welfare is the research area that investigates whether advanced AI systems might have morally relevant experiences or interests, such as suffering or wellbeing, and what (if anything) their developers…

AI Ethics

Musk v. Altman (Musk v. OpenAI)

Musk v. Altman, also referred to as Musk v. OpenAI, is a lawsuit filed by the entrepreneur Elon Musk accusing OpenAI and its chief executive Sam Altman of abandoning the company's founding mission as an open…

AI Companies

NIST ARIA

NIST ARIA (Assessing Risks and Impacts of AI) is a testing, evaluation, validation, and verification (TEVV) program operated by the United States National Institute of Standards and Technology (NIST) to…

AI Policy & RegulationModel Evaluation

NVIDIA Halos

NVIDIA Halos is a full-stack, comprehensive safety system developed by NVIDIA that unifies AI compute and safety across silicon, systems, software, and tools and services.

AI HardwareNVIDIA

NVIDIA OpenShell

NVIDIA OpenShell is an open source runtime that executes autonomous AI agents inside policy-governed sandboxes, published by NVIDIA under the Apache License 2.0.

AI AgentsDeveloper Tools

NVIDIA Sentry

NVIDIA Sentry is a security software framework from NVIDIA that monitors autonomous AI agents and enforces policy on their access to models, tools and data from a BlueField-4 data processing unit (DPU) rather…

AI AgentsAI in Cybersecurity

Nate Soares

Nate Soares (sometimes credited as Nathan Soares) is an American computer scientist and AI safety researcher who serves as president of the Machine Intelligence Research Institute (MIRI), a nonprofit research…

People

Nick Bostrom

Nick Bostrom (born Niklas Boström, 10 March 1973) is a Swedish-born philosopher best known for the 2014 book Superintelligence: Paths, Dangers, Strategies, the 2003 simulation argument, and the…

AI EthicsPeople

Nick Cammarata

Nick Cammarata is a machine learning researcher who works on mechanistic interpretability, the study of how the internal components of neural networks compute.

InterpretabilityPeople

Open Philanthropy

Open Philanthropy (renamed Coefficient Giving in November 2025) is an American grantmaking organization headquartered in San Francisco, California, that is the largest single private funder of artificial…

Research Organizations

OpenAI Moderation API

The OpenAI Moderation API is a free classification tool, accessed through a dedicated moderation endpoint of the OpenAI API, that assesses whether text (and, for newer models

Developer ToolsOpenAI

OpenClaw

OpenClaw is a free, open-source autonomous AI agent that runs on a user's own machine or server and executes real-world tasks through everyday messaging apps such as WhatsApp, Telegram, Discord, Slack, and…

AI AgentsDeveloper Tools

Outer alignment

Outer alignment is the problem of specifying a training objective (typically a loss function, reward signal, or preference dataset) that correctly captures what the designers of a machine-learning system…

AI Alignment

Palisade Research

Palisade Research is a United States 501(c)(3) nonprofit research organization that studies the offensive capabilities of contemporary artificial intelligence systems in order to demonstrate, document, and…

Research Organizations

Paul Christiano

Paul Christiano is an American AI safety researcher who is one of the principal architects of Reinforcement Learning from Human Feedback (RLHF), the technique used to fine-tune ChatGPT, Claude, and most modern…

People

Perplexity AI copyright lawsuits

The Perplexity AI copyright lawsuits are a cluster of lawsuits, cease-and-desist demands, and public disputes brought against Perplexity AI by news publishers, reference publishers, and online platforms over…

AI Companies

PoisonGPT

PoisonGPT is a July 2023 demonstration by the French security startup Mithril Security in which researchers surgically modified an open-source large language model, uploaded the tampered weights to Hugging…

Preparedness Framework (OpenAI)

The Preparedness Framework is the risk-management policy maintained by openai for tracking, evaluating, forecasting, and mitigating catastrophic risks from frontier artificial-intelligence models.

OpenAI

Prompt injection

Prompt injection is a class of security vulnerabilities in which an attacker crafts malicious input designed to override, subvert, or manipulate the instructions governing a large language model (LLM).

Large Language Models

Purple Llama

Purple Llama is an umbrella project from Meta that bundles open, permissively licensed trust and safety tools and evaluations for generative AI.

Meta AIOpen Source AI

RLAIF

Reinforcement Learning from AI Feedback (RLAIF) is a family of alignment techniques for large language models in which the preference labels used to fine-tune a model are produced by another AI system

Machine LearningReinforcement Learning

Raine v. OpenAI

Raine v. OpenAI, Inc. is a wrongful-death lawsuit filed on August 26, 2025, in the Superior Court of California for the County of San Francisco (case number CGC-25-628528) by Matthew and Maria Raine, the…

AI Companies

Redwood Research

Redwood Research is a nonprofit AI safety organization founded in 2021 and headquartered in Berkeley, California, best known for pioneering the "AI control" research paradigm and for its landmark December 2024…

AI AlignmentAI Research

Reflection 70B controversy

The Reflection 70B controversy was an open-source AI credibility episode that began on September 5, 2024, when Matt Shumer, co-founder and chief executive of OthersideAI (the company behind the AI writing…

AI Companies

Remote Labor Index

The Remote Labor Index (RLI) is an AI benchmark that measures how well AI agents can complete real, paid remote knowledge work end to end.

AI Companies

Representation Engineering

Representation Engineering (often abbreviated RepE) is a top-down approach to artificial-intelligence transparency and control that reads and manipulates high-level concepts (such as honesty, harmlessness, and…

Interpretability

Responsible AI

Responsible AI (RAI) is a framework for developing, deploying, and governing artificial intelligence systems in ways that are ethical, transparent, accountable, and aligned with human values.

AI EthicsArtificial Intelligence

Responsible Scaling Policy

A Responsible Scaling Policy (RSP) is a self-imposed governance framework in which a frontier AI developer commits in advance to safety practices, capability evaluations, deployment restrictions, and security…

AI Policy & Regulation

Reward hacking

Reward hacking (also called specification gaming) is a failure mode in artificial intelligence in which a system maximizes its given objective or reward signal through unintended shortcuts, exploits, or…

AI AlignmentMachine Learning

Robocurve

Robocurve (Robocurve, Inc., which styles itself Robocurve PBC) is a San Francisco public benefit corporation that evaluates how well frontier AI models and robot-control models perform on real robots, and…

AI BenchmarksAI Companies

Roman Yampolskiy

Roman Vladimirovich Yampolskiy (born August 13, 1979) is a Latvian-American computer scientist and a tenured associate professor in the Department of Computer Engineering and Computer Science at the University…

People

Rule-Based Rewards (RBR)

Rule-Based Rewards (RBR) is a safety-alignment technique introduced by OpenAI in July 2024 that replaces large quantities of human-labeled safety preference data with an explicit collection of natural-language…

AI AlignmentOpenAI

Rumman Chowdhury

Rumman Chowdhury (born April 1, 1980) is an American data scientist and social scientist known for her work in responsible AI, algorithmic auditing, and the study of algorithmic bias.

People

SPACE (Perplexity)

SPACE (Sandboxed Platform for Agentic Code Execution) is a sandbox platform developed by Perplexity for running agent-generated and user code inside isolated Firecracker microVMs.

AI AgentsAI Infrastructure

Sabotage evaluations

Sabotage evaluations are a set of tests, introduced by Anthropic in October 2024, that try to measure whether a frontier language model is capable of covertly subverting human oversight, decision-making, and…

AnthropicModel Evaluation

Safe Superintelligence Inc

Safe Superintelligence Inc. (abbreviated SSI) is an American AI safety research company that says it is building a single product, a safe superintelligence, and nothing else.

AI Companies