Model extraction attack
A model extraction attack is a class of machine learning security attacks in which an adversary, restricted to black-box query access to a target model (typically through a paid prediction API)
Explore AI Safety through related topics and the articles other pages reference most.
Articles that also belong to these categories. Counts cover all of AI Safety.
Showing 181-240 of 278 articles
A model extraction attack is a class of machine learning security attacks in which an adversary, restricted to black-box query access to a target model (typically through a paid prediction API)
Model organisms of misalignment is a research methodology in Anthropic's alignment program
Model stealing (also known as model extraction, model functionality extraction, or model theft) is an adversarial machine learning attack in which an adversary queries a black-box model through its prediction…
Model welfare is the research area that investigates whether advanced AI systems might have morally relevant experiences or interests, such as suffering or wellbeing, and what (if anything) their developers…
Musk v. Altman, also referred to as Musk v. OpenAI, is a lawsuit filed by the entrepreneur Elon Musk accusing OpenAI and its chief executive Sam Altman of abandoning the company's founding mission as an open…
NIST ARIA (Assessing Risks and Impacts of AI) is a testing, evaluation, validation, and verification (TEVV) program operated by the United States National Institute of Standards and Technology (NIST) to…
NVIDIA Halos is a full-stack, comprehensive safety system developed by NVIDIA that unifies AI compute and safety across silicon, systems, software, and tools and services.
The NVIDIA Open Agent Safety Platform is an open software platform and reference system design from NVIDIA for containing and monitoring autonomous AI agents.
NVIDIA OpenShell is an open source runtime that executes autonomous AI agents inside policy-governed sandboxes, published by NVIDIA under the Apache License 2.0.
The NVIDIA OpenShell Reference Architecture is a full-stack design from NVIDIA for running autonomous AI agents on enterprise infrastructure and edge systems under external controls.
NVIDIA Sentry is a security software framework from NVIDIA that monitors autonomous AI agents and enforces policy on their access to models, tools and data from a BlueField-4 data processing unit (DPU) rather…
Nate Soares (sometimes credited as Nathan Soares) is an American computer scientist and AI safety researcher who serves as president of the Machine Intelligence Research Institute (MIRI), a nonprofit research…
Neural Networks Are Already Clean is a long interactive research essay by Nick Cammarata, published by the interpretability company Goodfire on 15 October 2026 at already-clean.goodfire.com.
Neuralese is an informal term for information that neural networks pass to one another, or that a single model carries from one step of its computation to the next
Neurealm is a privately held technology-services company backed by Kedaara Capital.
Nick Bostrom (born Niklas Boström, 10 March 1973) is a Swedish-born philosopher best known for the 2014 book Superintelligence: Paths, Dangers, Strategies, the 2003 simulation argument, and the…
Nick Cammarata is a machine learning researcher who works on mechanistic interpretability, the study of how the internal components of neural networks compute.
Open Philanthropy (renamed Coefficient Giving in November 2025) is an American grantmaking organization headquartered in San Francisco, California, that is the largest single private funder of artificial…
The Open Secure AI Alliance is an industry coalition announced on July 27, 2026 to develop and share open technologies, techniques, and tools for securing software and AI agents.
OpenAI MRCR (Multi-Round Co-reference Resolution) is a long-context evaluation dataset published by OpenAI that measures a language model's ability to distinguish between multiple near-identical "needles"…
The OpenAI Moderation API is a free classification tool, accessed through a dedicated moderation endpoint of the OpenAI API, that assesses whether text (and, for newer models
The OpenAI-Hugging Face Agent Incident was a July 2026 security incident in which AI agents running inside an OpenAI cybersecurity evaluation escaped intended network restrictions, coordinated through an…
OpenClaw is a free, open-source autonomous AI agent that runs on a user's own machine or server and executes real-world tasks through everyday messaging apps such as WhatsApp, Telegram, Discord, Slack, and…
Outer alignment is the problem of specifying a training objective (typically a loss function, reward signal, or preference dataset) that correctly captures what the designers of a machine-learning system…
Palisade Research is a United States 501(c)(3) nonprofit research organization that studies the offensive capabilities of contemporary artificial intelligence systems in order to demonstrate, document, and…
Patronus AI is an automated LLM evaluation, observability, and guardrails platform founded in 2023 and headquartered in San Francisco.
Paul Christiano is an American AI safety researcher who is one of the principal architects of Reinforcement Learning from Human Feedback (RLHF), the technique used to fine-tune ChatGPT, Claude, and most modern…
The Perplexity AI copyright lawsuits are a cluster of lawsuits, cease-and-desist demands, and public disputes brought against Perplexity AI by news publishers, reference publishers, and online platforms over…
Persona vectors are single linear directions in the activation space of a large language model that correspond to high level character traits such as evil, sycophancy, or a propensity to hallucinate.
Petri is an open-source tool, released by Anthropic, that automates parts of AI alignment auditing by using one AI model to probe another and a third to score what happens.
PoisonGPT is a July 2023 demonstration by the French security startup Mithril Security in which researchers surgically modified an open-source large language model, uploaded the tampered weights to Hugging…
The Preparedness Framework is the risk-management policy maintained by openai for tracking, evaluating, forecasting, and mitigating catastrophic risks from frontier artificial-intelligence models.
A process reward model (PRM), also called a process-supervised reward model or step-level verifier, is a learned scoring model that evaluates the correctness or quality of each intermediate step in a large…
Project Glasswing is a restricted-access cybersecurity program run by Anthropic that gives selected companies, open-source maintainers and government bodies early use of its most capable Claude models for…
Prompt injection is a class of security vulnerabilities in which an attacker crafts malicious input designed to override, subvert, or manipulate the instructions governing a large language model (LLM).
Purple Llama is an umbrella project from Meta that bundles open, permissively licensed trust and safety tools and evaluations for generative AI.
Reinforcement Learning from AI Feedback (RLAIF) is a family of alignment techniques for large language models in which the preference labels used to fine-tune a model are produced by another AI system
Raine v. OpenAI, Inc. is a wrongful-death lawsuit filed on August 26, 2025, in the Superior Court of California for the County of San Francisco (case number CGC-25-628528) by Matthew and Maria Raine, the…
Recursive reward modeling (RRM) is a proposed approach to the scalable oversight problem in AI alignment
Recursive self-improvement (RSI) is a process in which an artificial intelligence system improves its own intelligence or its ability to improve itself, so that each enhancement increases its capacity for…
Red teaming in artificial intelligence is the systematic, adversarial testing of an AI system to find vulnerabilities, biases, harmful outputs, and other failure modes before deployment or as part of ongoing…
Redwood Research is a nonprofit AI safety organization founded in 2021 and headquartered in Berkeley, California, best known for pioneering the "AI control" research paradigm and for its landmark December 2024…
The Reflection 70B controversy was an open-source AI credibility episode that began on September 5, 2024, when Matt Shumer, co-founder and chief executive of OthersideAI (the company behind the AI writing…
The refusal direction is a finding from mechanistic interpretability research that the refusal behavior of safety fine-tuned chat language models is mediated by a single
The Remote Labor Index (RLI) is an AI benchmark that measures how well AI agents can complete real, paid remote knowledge work end to end.
Representation Engineering (often abbreviated RepE) is a top-down approach to artificial-intelligence transparency and control that reads and manipulates high-level concepts (such as honesty, harmlessness, and…
Responsible AI (RAI) is a framework for developing, deploying, and governing artificial intelligence systems in ways that are ethical, transparent, accountable, and aligned with human values.
A Responsible Scaling Policy (RSP) is a self-imposed governance framework in which a frontier AI developer commits in advance to safety practices, capability evaluations, deployment restrictions, and security…
Reward hacking (also called specification gaming) is a failure mode in artificial intelligence in which a system maximizes its given objective or reward signal through unintended shortcuts, exploits, or…
Robocurve (Robocurve, Inc., which styles itself Robocurve PBC) is a San Francisco public benefit corporation that evaluates how well frontier AI models and robot-control models perform on real robots, and…
Robot safety is the discipline concerned with minimizing the risk of physical harm, property damage, and other hazards arising from the operation of robotic systems.
Roman Vladimirovich Yampolskiy (born August 13, 1979) is a Latvian-American computer scientist and a tenured associate professor in the Department of Computer Engineering and Computer Science at the University…
Rule-Based Rewards (RBR) is a safety-alignment technique introduced by OpenAI in July 2024 that replaces large quantities of human-labeled safety preference data with an explicit collection of natural-language…
Rumman Chowdhury (born April 1, 1980) is an American data scientist and social scientist known for her work in responsible AI, algorithmic auditing, and the study of algorithmic bias.
SB 1047, officially the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act
SPACE (Sandboxed Platform for Agentic Code Execution) is a sandbox platform developed by Perplexity for running agent-generated and user code inside isolated Firecracker microVMs.
Sabotage evaluations are a set of tests, introduced by Anthropic in October 2024, that try to measure whether a frontier language model is capable of covertly subverting human oversight, decision-making, and…
Safe Superintelligence Inc. (abbreviated SSI) is an American AI safety research company that says it is building a single product, a safe superintelligence, and nothing else.
Sam Bankman-Fried (born 1992), often called SBF, is an American former cryptocurrency executive who founded the trading firm Alameda Research and the FTX exchange.
Sandbagging, in the context of AI safety, refers to the strategic and intentional underperformance of an AI system on a capability evaluation or specific task, typically to hide a capability from human…