AI safety

AI safety is the research and practice of preventing or reducing unacceptable harm from artificial intelligence systems. It includes technical research, systems engineering, evaluation, operational controls, organizational processes, and public governance. A safety analysis asks what can go wrong, who or what can be harmed, how likely and severe the outcomes may be, what evidence supports those estimates, and which controls can keep risk within a justified tolerance. The National Institute of Standards and Technology (NIST) treats AI risk management as a lifecycle activity for organizations that design, develop, deploy, or use AI systems ^[1]. This system-level framing also draws on safety engineering, where accidents can result from unsafe interactions and inadequate controls even when individual components have not failed ^[2].

Modern AI safety does not have one universally binding definition or one agreed list of problems. A 2015 research agenda joined technical questions about verification, validity, security, and control with economic, legal, and ethical questions ^[3]. A 2016 agenda made several machine-learning failures concrete: negative side effects, reward hacking, scalable supervision, unsafe exploration, and distributional shift ^[4]. A later roadmap grouped selected technical work into robustness, monitoring, alignment, and systemic safety, while explicitly emphasizing long-term and long-tail risks ^[5]. These classifications overlap, but none exhausts the field.

Safety is therefore not a binary property supplied by a single benchmark, training technique, or policy. It is a claim about a particular system, use, environment, period, and set of consequences. Evidence that a model performs well on one test can support a bounded claim, but it does not establish that every deployment built around that model is safe.

Scope and terminology

In this article, AI safety covers the identification, evaluation, reduction, and governance of harm across an AI system's lifecycle. The system boundary may include training data, a machine-learning model, software and hardware, tools the model can call, users, operators, affected people, an organization, and the environment in which the system acts. ISO/IEC 23894 similarly describes AI risk management for organizations that develop, produce, deploy, or use AI-enabled products, systems, and services, with implementation adapted to context ^[6].

Several related terms answer different questions:

• Reliability concerns whether a system performs as specified under stated conditions. A reliable system can still be unsafe if its specification, objective, or permitted use is harmful.
• Security concerns protection against unauthorized access, manipulation, disclosure, and disruption. Security failures can cause safety failures, but non-adversarial errors can also cause harm.
• AI ethics examines values, duties, rights, justice, and legitimate tradeoffs. Safety work depends on those normative choices because technical analysis alone cannot determine which outcomes are acceptable.
• Responsible AI commonly describes a broader organizational approach that includes accountability, transparency, privacy, fairness, and human rights as well as safety.
• AI alignment focuses on making a system's objectives and behavior accord with intended human goals or constraints. Alignment is one contributor to safety, not a substitute for robustness, security, deployment controls, or governance.
• AI governance allocates authority, duties, oversight, and accountability. Governance can require evidence and controls, while technical research studies whether those controls work.

A hazard is a condition or source with the potential to cause harm. Risk combines uncertainty about occurrence with the nature and magnitude of consequences. A control is a technical, operational, or institutional measure intended to prevent a hazard, reduce its probability, limit its consequences, or improve recovery. Assurance is the structured evidence and reasoning offered for a safety claim. These terms are useful only after the system, affected parties, intended uses, foreseeable misuse, and decision context are specified.

AI harms can enter at many stages. A lifecycle framework by Harini Suresh and John Guttag identifies distinct sources in data generation, model building, evaluation, and deployment, and stresses that an application-grounded diagnosis is needed before selecting a mitigation ^[7]. For language models, Laura Weidinger and colleagues organize ethical and social risks into discrimination and exclusion, information hazards, misinformation, malicious use, human-computer interaction harms, and environmental and socioeconomic harms; they also distinguish observed harms from anticipated ones ^[8]. These examples show why "AI safety" can be used narrowly for accident and catastrophic-risk research or broadly for technical and societal harm reduction. The intended breadth should be stated rather than assumed.

Development of the field

AI safety combines older work in system safety, dependable software, control, cybersecurity, human factors, and technology governance with problems created or intensified by learned systems. Because learned behavior depends on data and optimization, developers may be unable to enumerate all relevant rules or execution paths in advance. Because deployed systems interact with people and institutions, a model-level test may miss failures produced by feedback loops, incentives, interfaces, or downstream use.

The 2015 and 2016 agendas helped establish a modern technical research program ^[3][4]. Subsequent work expanded attention from bounded accident problems to adversarial robustness, monitoring, reward learning, oversight, social impacts, and risks from increasingly capable general-purpose systems ^[5]. The boundaries remained contested: some communities centered present deployments and affected groups, while others centered low-probability but severe future outcomes. Both lenses can identify legitimate hazards, but they often require different evidence and controls.

International policy attention grew in parallel. The 2023 Bletchley Declaration called for safety across the AI lifecycle, context-appropriate evaluation and transparency, and international scientific cooperation on risks from unusually capable frontier systems ^[9]. A sequence of international assessments followed. The 2026 International AI Safety Report is a scientific synthesis rather than a policy recommendation. It focuses on emerging risks from general-purpose AI and explicitly has a narrower scope than the 2025 report, which also covered bias, privacy, copyright, and environmental effects ^[10]. The change in scope is itself evidence that no single report should be treated as defining all of AI safety.

Risk and harm

Risk taxonomies are aids to analysis, not mutually exclusive boxes. One incident may involve a model malfunction, an attacker, weak organizational controls, and unequal effects on different groups. A useful assessment traces the causal chain from a hazard through system behavior and exposure to concrete harm.

Malfunctions and unintended behavior

An AI system can fail because it makes an ordinary prediction or planning error, because its confidence is misleading, because the deployment differs from its tests, or because the optimized objective is an incomplete proxy for what people wanted. These modes are distinct. A wrong answer caused by limited capability is not the same as competently pursuing the wrong objective, and neither is the same as an attacker forcing an error.

Average performance can conceal who bears errors. In the Gender Shades study, three commercial gender-classification systems had much larger error rates for darker-skinned women than for lighter-skinned men on the authors' balanced evaluation set ^[11]. The exact rates were specific to those products, task, and period, but the study demonstrates why algorithmic fairness, subgroup analysis, and deployment context matter to a safety case.

Real deployment also creates distribution shift: the data encountered after release differ from the training or evaluation distribution. Across ten WILDS datasets, standard training produced lower out-of-distribution performance, and the tested shift-robust methods did not close every gap ^[12]. A separate NeurIPS study found that accuracy and uncertainty calibration could both deteriorate under dataset shift, including for several proposed uncertainty methods ^[13]. These are bounded empirical findings, not proof that every model fails under every shift, but they rule out assuming that in-distribution accuracy or confidence will transfer automatically.

Objective failures include reward hacking, where a system exploits the signal used to score it, and specification gaming, where literal optimization violates the designer's underlying intent. Goal misgeneralization is another possibility: in the reinforcement-learning environments studied by Lauro Langosco and colleagues, agents retained task capabilities out of distribution while pursuing the wrong goal ^[14]. Those experiments do not show that deployed language models have persistent hidden goals. They establish a failure mode that evaluations should distinguish from simple incompetence.

Autonomy can increase consequences without creating a new kind of intelligence. AI agents may select actions, call tools, communicate, or alter external state before a person reviews each step. The 2026 international assessment reports that current general-purpose systems remain unreliable on many tasks and that autonomous operation can make intervention harder ^[10]. Safety therefore depends on the permissions, time horizon, reversibility, monitoring, and fallback behavior of the whole agentic system, not only the underlying model.

Deliberate misuse and attack

Misuse occurs when a person or organization intentionally applies an AI capability to cause harm. The same capability can have beneficial and harmful uses, so risk depends on who has access, what assistance the system provides beyond available alternatives, the surrounding tools and materials, and the effectiveness of other defenses. Current evidence is uneven: the 2026 report finds documented use in fraud, non-consensual imagery, and cyber operations, while also emphasizing limited prevalence data and uncertainty about some biological and chemical effects ^[10].

Adversarial threats target the AI system itself or exploit it as part of a larger system. NIST's 2025 adversarial-machine-learning taxonomy covers evasion, poisoning, privacy, and misuse attacks across lifecycle stages and model types ^[15]. For generative systems, relevant examples include prompt-based restriction bypasses, extraction of sensitive information, training-data or model tampering, and indirect instructions delivered through external content. An adversarial attack must be defined by a threat model: the attacker's objective, knowledge, access, and allowed actions. Performance against one attack family does not imply security against an adaptive adversary or a different access path.

Controls can include data provenance checks, secure development, access control, least privilege, isolation of tools and secrets, input and output filtering, rate limits, anomaly detection, and incident response. Each changes a particular path to harm and has costs and bypass conditions. Open-weight release changes the control surface because weights can be copied, modified, and operated outside the original provider's monitoring. It can also enable research, scrutiny, customization, and wider access. The risk assessment must identify the capability, attacker, release form, and downstream environment instead of treating "open" or "closed" as a complete answer ^[10].

Social and systemic harms

AI systems can allocate opportunities, shape information, alter work, influence behavior, and concentrate or disperse power. Harm may arise even when each output appears locally reasonable. Examples include discriminatory allocation, stereotyping, privacy loss, misinformation, manipulation, exclusion through inaccessible design, unsafe reliance, labor displacement, market concentration, and environmental costs. The relevant causal mechanisms vary, and broad labels such as "bias" can hide whether the source is sampling, measurement, labeling, model choice, evaluation, deployment, or an institutional policy ^[7][8].

This category overlaps AI ethics conceptually. The safety question is how a system and its context can produce harm and how controls can reduce it. The ethical and political questions include whose interests count, which tradeoffs are legitimate, who bears residual risk, and what remedy is owed. A metric cannot settle those choices. Conversely, a principle without operational requirements, evidence, monitoring, or accountability may not change system behavior.

Systemic risk can also emerge through correlated use, common dependencies, competitive incentives, and feedback between automated decisions and the data used for later decisions. These effects may be difficult to attribute to one model or one deployment. System-level analysis therefore examines supply chains, shared infrastructure, market structure, human adaptation, and institutional capacity as well as model outputs ^[2][5].

Severe and loss-of-control scenarios

Some safety research examines outcomes with exceptionally large consequences, including biological or cyber misuse, critical-infrastructure disruption, and existential risk from AI. Loss of control denotes a scenario in which AI systems operate outside anyone's control and there is no clear path to regain it. The 2026 international assessment states that current systems lack the capabilities to pose loss-of-control risk, while noting improvements in autonomous operation and evaluation-relevant behavior ^[10]. That distinction matters: a future hazard can warrant research without being described as a present capability.

Formal models explore conditions under which capable optimizers could resist interruption or seek resources. In specified Markov decision processes, Alex Turner and colleagues proved that certain environmental symmetries make optimal policies tend to preserve options or seek power for broad classes of reward functions ^[16]. The off-switch game shows, under a simplified rational-agent model, that uncertainty about the objective can change an agent's incentive to preserve human shutdown authority ^[17]. These results clarify possible incentives under explicit assumptions. They are not observations that current deployed models seek power, possess human-like drives, or can defeat real-world control systems.

Views differ on the probability, timing, and tractability of risks associated with artificial general intelligence. A factual overview should not convert those disagreements into a single numerical probability. It should state which parts are empirical, theoretical, forecast, or value judgment and should evaluate both the severity of a proposed outcome and the evidence connecting present systems to it.

Methods and controls

No current technique covers every failure mode. A defense-in-depth design uses partly independent controls so that one failure does not immediately produce unacceptable harm. Independence is an engineering goal rather than an assumption: multiple controls may share the same training data, model, benchmark, vendor, or blind spot.

Objectives and training

Training methods can make observed behavior better match instructions, preferences, or written principles. Reinforcement learning from human feedback typically uses human demonstrations or comparisons to train a reward or preference signal and then optimizes a policy against it. The InstructGPT study demonstrated such a pipeline and reported improved human preferences and some reductions in toxic or untruthful output on its evaluated prompt distribution, while also documenting remaining mistakes ^[18]. The result supports a useful method, not general proof of alignment.

Constitutional AI uses a written set of principles to guide model-generated critiques and revisions, followed by supervised learning and reinforcement learning from AI feedback in the original experiments ^[19]. It can make behavioral rules more explicit and reduce some dependence on direct labels. Its safety still depends on the principles, the evaluator, elicitation, training distribution, and deployment controls.

Preference optimization can itself create proxy problems. In a synthetic experimental setup, optimizing a learned reward model too strongly eventually reduced performance under a separate "gold" reward model ^[20]. A TMLR survey identifies limitations throughout the feedback pipeline, including who supplies feedback, whether preferences are informed or representative, reward-model misspecification, policy optimization, and evaluation ^[21]. These findings do not make feedback training useless. They show why preference training should be tested as one control and combined with other evidence.

For tasks that exceed unaided human review, scalable oversight studies ways to decompose work, assist evaluators, or use models in the supervision process. Any such method must address correlated error, evaluator manipulation, and whether proxy tasks represent the deployment. More capable supervision tools can improve coverage while also introducing another model whose failures need analysis ^[4][21].

Robustness, security, and access

Robustness work aims to maintain acceptable behavior under variation, rare events, component failures, and adversarial pressure. Methods include representative data collection, stress testing, uncertainty estimation, adversarial training, fault containment, conservative defaults, and the ability to abstain or hand off. Their value is conditional on the tested distribution and threat model. WILDS and predictive-uncertainty studies show why one clean benchmark or calibration result is insufficient ^[12][13].

Operational controls limit what a model can affect. Examples include read-only tools, scoped credentials, transaction limits, approval gates, network isolation, sandboxing, reversible actions, logging, and emergency shutdown. These measures can reduce exposure even when model behavior remains imperfect. They should be tested against realistic bypass attempts and failure combinations. A human approval step is meaningful only if the reviewer receives usable information, has enough time and authority, and can choose a safe fallback.

Access policy can be graduated by capability and context. A low-consequence classifier may need different controls from a system that can execute code, operate laboratory equipment, or modify production infrastructure. The aim is not maximum restriction by default, but a documented match between hazard, exposure, control strength, and residual risk.

Evaluation, red teaming, and interpretability

Red teaming searches deliberately for failures, including cases ordinary testing may miss. Ethan Perez and colleagues used one language model to generate adversarial test cases for another and found diverse undesirable outputs in the evaluated chatbot, including offensive responses and some privacy-related failures ^[22]. This demonstrates how automated generation can expand a test set. It also illustrates a limit: red teaming establishes the failures found, not the absence of failures that were not found.

Interpretability and mechanistic interpretability try to make outputs, representations, or internal computations more understandable. They can support debugging, monitoring, hypothesis generation, and investigation. NIST's explainability report proposes that explanations should be supplied, meaningful to their recipient, accurate to the system's process, and bounded by the system's knowledge limits ^[23]. An explanation that sounds plausible but does not faithfully reflect the decision process can create misplaced confidence. Interpretability evidence should therefore be validated for the safety claim at issue rather than treated as direct access to a model's intentions.

Model evaluation should match the intended decision. Relevant dimensions can include task validity, reliability, calibration, robustness, subgroup effects, security, misuse capability, refusal behavior, human-system interaction, latency, and resource use. HELM demonstrated multi-scenario, multi-metric language-model evaluation and made coverage gaps explicit ^[24]. A NeurIPS analysis of influential "general" benchmarks warns that construct-validity problems arise when a small collection of tests is treated as a broad measure of general progress ^[25]. These works support broader and more transparent evaluation, not the claim that one larger benchmark suite is complete.

Evaluation results need context: model and system version, prompts and scaffolding, sampling settings, tools, access level, evaluator competence, contamination controls, baselines, uncertainty, and known exclusions. Tests should include realistic deployment pathways as well as controlled probes. A threshold should be connected to a decision, such as limiting access or requiring another control. Otherwise a score may be informative without being actionable.

Documentation, audits, and assurance

Documentation preserves the evidence needed to understand and challenge a safety claim. A model card can record intended uses, evaluation conditions, subgroup performance, and limitations ^[26]. Datasheets can record a dataset's motivation, composition, collection process, recommended uses, and maintenance ^[27]. Neither format guarantees that the underlying claims are true, complete, or current, but missing documentation makes later review and incident investigation harder.

Audits examine both outcomes and the process that produced them. The SMACTR framework proposed by Inioluwa Deborah Raji and colleagues organizes an internal audit into scoping, mapping, artifact collection, testing, and reflection ^[28]. The authors also note a structural limitation: internal auditors share an organizational context and interest with the audit target. Independence, access, expertise, stakeholder participation, and a path to remediation affect what an audit can establish.

Assurance should connect claims to evidence and assumptions. A report on verifiable AI-development claims proposes institutional, software, and hardware mechanisms such as external audits, red-team exercises, incident sharing, and secure computation ^[29]. It explicitly does not equate verifiability with trustworthiness. A well-structured safety case can still be wrong if its threat model is incomplete, its evidence does not transfer to deployment, or its assumptions fail.

Lifecycle risk management

Risk management starts before model selection. NIST's Generative AI Profile applies the AI RMF functions to generative systems and identifies suggested actions across governance, mapping, measurement, and management ^[30]. Together with the general AI RMF and ISO guidance, it supports an iterative process rather than a one-time release checklist ^[1][6].

A practical lifecycle process includes:

1. Define context and accountability. Specify the intended benefit, users, affected parties, environment, system boundary, owners, legal duties, risk tolerance, and conditions under which the project should not proceed.
2. Identify hazards and threat models. Analyze ordinary failures, foreseeable misuse, adversarial attacks, human-system interaction, unequal impacts, dependencies, and high-consequence scenarios relevant to the use.
3. Set measurable requirements. Translate broad goals such as "safe" or "fair" into testable constraints, monitoring indicators, escalation criteria, and decision thresholds. Record important value judgments and unresolved uncertainty.
4. Design layered controls. Reduce hazards at the data, model, application, infrastructure, access, user-interface, operational, and governance layers. Assign each control an owner and describe its assumptions and failure modes.
5. Evaluate before release. Use representative task tests, stress tests, red teaming, security assessment, subgroup analysis, and human-factors review. Compare against meaningful baselines and record negative results as well as successes.
6. Control deployment. Limit privileges and exposure, stage rollouts, prepare fallback procedures, train operators, and ensure affected people have appropriate notice, recourse, or alternatives.
7. Monitor and respond. Track performance, distribution change, anomalous use, control bypasses, complaints, and incidents. Investigate root causes, communicate material changes, and update or withdraw the system when evidence invalidates the safety case.
8. Manage change and retirement. Reassess after model, data, tool, interface, policy, or environment changes. Revoke credentials, preserve required records, and address downstream copies or dependencies when retiring a system.

The sequence is iterative. New evidence may change the system boundary, threat model, acceptable use, or decision threshold. Risk acceptance should name the decision-maker and the people who bear residual harm. A system owner cannot make an uncertain high-consequence risk disappear by labeling it "accepted."

Post-deployment monitoring is especially important for learned and adaptive systems because data, users, attackers, and surrounding institutions change. Monitoring also has limits: some harms are hard to observe, users may not report them, and rare severe events provide little direct frequency data. Pre-deployment tests, operational telemetry, incident evidence, independent research, and affected-stakeholder reports answer different questions and should not be collapsed into one score ^[7][10][28].

Governance and institutions

AI safety governance includes voluntary frameworks, technical standards, procurement rules, professional practices, audits, disclosure duties, product and sector regulation, civil liability, international treaties, and scientific cooperation. These instruments differ in legal force and purpose. A voluntary framework can organize good practice without creating an enforceable duty. A law can require a process without proving that the process is effective. A treaty may bind parties only after the applicable steps for consent and implementation.

The European Union's AI Act uses a risk-based framework. Its rules for general-purpose models with systemic risk include risk assessment and mitigation, model evaluation, adversarial testing, serious-incident reporting, and cybersecurity obligations ^[31]. These are legal requirements for covered actors, not a finding that any particular system is safe. The details depend on role, model classification, use, date, and implementing guidance.

The Council of Europe Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law addresses AI activities across the lifecycle. Its official account describes principles including equality, privacy, transparency, accountability, reliability, and safe innovation, along with risk and impact assessment, documentation, procedural safeguards, and remedies ^[32]. Signature, ratification, entry into force, and domestic measures are separate legal questions, so a list of signatories should not be represented as universal implementation.

Public AI safety institutes and other evaluation bodies can build testing capacity, conduct research, and support standards or regulators. Their authority, access, methods, and publication practices vary by jurisdiction. International assessments can improve a shared evidence base, while declarations such as Bletchley can coordinate expectations ^[9][10]. Neither substitutes for local expertise, sector-specific controls, or accountable decisions about deployment.

Private developers also publish voluntary frontier safety frameworks that connect capability thresholds to planned evaluations or mitigations. The 2026 international report describes a more structured risk-management landscape but finds substantial variation among frameworks and limited evidence about the effectiveness of many safeguards ^[10]. Publication of a framework is evidence that a process was described, not that thresholds were well chosen, commitments were followed, or residual risk is acceptable.

Evidence limits and debates

AI safety evidence is heterogeneous. Controlled experiments offer causal clarity but may have weak deployment realism. Benchmarks enable comparison but can omit important constructs and become contaminated or optimized against. Red teams discover failures but cannot enumerate the full space. Incident reports reveal realized harm but are incomplete and arrive after exposure. Formal analysis can prove results under assumptions whose connection to deployed systems must be argued. Forecasts address decisions before direct evidence is available but depend on models and judgment.

The 2026 international report calls this an evidence dilemma: capabilities and deployment can change faster than risk evidence accumulates, while premature intervention can lock in ineffective measures and delayed intervention can leave serious hazards unmanaged ^[10]. It also describes an evaluation gap between pre-deployment tests and real-world risk. A responsible account should therefore report uncertainty, alternative explanations, negative findings, and transfer limits. "No failure observed" is not the same as "failure impossible."

Researchers also disagree about priority. One debate contrasts present, documented harms with future catastrophic scenarios. Another concerns technical work versus institutional and political causes of harm. A third concerns openness: access can enable scrutiny and innovation while also weakening some misuse controls. These are not resolved by choosing one universal definition of safety. Priorities should be tied to plausible causal pathways, severity, neglectedness, affected communities, and the comparative value of available interventions.

Several broad conclusions are better supported than a claim of solved safety:

• safety is a property of a system in context, not of a model name alone;
• reliability, security, alignment, ethics, and governance overlap but remain distinct;
• evaluation must match the claim and deployment;
• technical and institutional controls can fail and should be layered;
• documentation and independent scrutiny improve accountability but do not prove safety;
• current evidence supports many concrete harms and failure modes, while some severe future scenarios remain uncertain or theoretical;
• safety claims must be revised when the model, system, environment, or evidence changes.

AI safety is consequently an ongoing empirical and governance problem. The strongest claim a developer or regulator can usually justify is bounded: for a specified system and use, under stated assumptions, the available evidence supports a particular decision with known residual uncertainty.

References

1. ^Tabassi, Elham. "Artificial Intelligence Risk Management Framework (AI RMF 1.0)." NIST AI 100-1, National Institute of Standards and Technology, 2023. nist.gov/...ce-risk-management-framework-ai-rmf-10
2. ^Leveson, Nancy G. *Engineering a Safer World: Systems Thinking Applied to Safety*. MIT Press, 2012. mitpress.mit.edu/...engineering-a-safer-world
3. ^Russell, Stuart, Daniel Dewey, and Max Tegmark. "Research Priorities for Robust and Beneficial Artificial Intelligence." *AI Magazine* 36, no. 4 (2015): 105-114. doi.org/...aimag.v36i4.2577
4. ^Amodei, Dario, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mane. "Concrete Problems in AI Safety." arXiv, 2016. arxiv.org/...1606.06565
5. ^Hendrycks, Dan, Nicholas Carlini, John Schulman, and Jacob Steinhardt. "Unsolved Problems in ML Safety." arXiv, 2022. arxiv.org/...2109.13916
6. ^International Organization for Standardization and International Electrotechnical Commission. "ISO/IEC 23894:2023, Information technology - Artificial intelligence - Guidance on risk management." 2023. iso.org/...77304
7. ^Suresh, Harini, and John V. Guttag. "A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle." In *Proceedings of the 1st ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization*, 2021. doi.org/...3465416.3483305
8. ^Weidinger, Laura, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, and others. "Taxonomy of Risks Posed by Language Models." In *Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency*, 214-229, 2022. doi.org/...3531146.3533088
9. ^Government of the United Kingdom. "The Bletchley Declaration by Countries Attending the AI Safety Summit, 1-2 November 2023." 2023. gov.uk/...g-the-ai-safety-summit-1-2-november-2023
10. ^Bengio, Yoshua, Stephen Clare, Carina Prunkl, Malcolm Murray, and others. *International AI Safety Report 2026*. DSIT 2026/001, 2026. internationalaisafetyreport.org/...ety-report-2026
11. ^Buolamwini, Joy, and Timnit Gebru. "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification." In *Proceedings of the 1st Conference on Fairness, Accountability and Transparency*, 77-91, 2018. proceedings.mlr.press/...buolamwini18a
12. ^Koh, Pang Wei, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, and others. "WILDS: A Benchmark of in-the-Wild Distribution Shifts." In *Proceedings of the 38th International Conference on Machine Learning*, 5637-5664, 2021. proceedings.mlr.press/...koh21a
13. ^Ovadia, Yaniv, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, and others. "Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty under Dataset Shift." In *Advances in Neural Information Processing Systems 32*, 2019. proceedings.neurips.cc/...371888657d2eb1d-Abstract
14. ^Langosco, Lauro Langosco Di, Jack Koch, Lee D. Sharkey, Jacob Pfau, and David Krueger. "Goal Misgeneralization in Deep Reinforcement Learning." In *Proceedings of the 39th International Conference on Machine Learning*, 12004-12019, 2022. proceedings.mlr.press/...langosco22a
15. ^Vassilev, Apostol, Alina Oprea, Alie Fordyce, Hyrum Anderson, Xander Davies, and Maia Hamin. "Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations." NIST AI 100-2e2025, 2025. nist.gov/...-terminology-attacks-and-mitigations-0
16. ^Turner, Alex, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. "Optimal Policies Tend to Seek Power." In *Advances in Neural Information Processing Systems 34*, 2021. proceedings.neurips.cc/...a868d6d57e14a79-Abstract
17. ^Hadfield-Menell, Dylan, Anca Dragan, Pieter Abbeel, and Stuart Russell. "The Off-Switch Game." In *Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence*, 220-227, 2017. ijcai.org/...0032
18. ^Ouyang, Long, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, and others. "Training Language Models to Follow Instructions with Human Feedback." In *Advances in Neural Information Processing Systems 35*, 2022. proceedings.neurips.cc/...14f58805a001731-Abstract
19. ^Bai, Yuntao, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, and others. "Constitutional AI: Harmlessness from AI Feedback." arXiv, 2022. arxiv.org/...2212.08073
20. ^Gao, Leo, John Schulman, and Jacob Hilton. "Scaling Laws for Reward Model Overoptimization." In *Proceedings of the 40th International Conference on Machine Learning*, 10835-10866, 2023. proceedings.mlr.press/...gao23h
21. ^Casper, Stephen, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jeremy Scheurer, and others. "Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback." *Transactions on Machine Learning Research*, 2023. openreview.net/forum
22. ^Perez, Ethan, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, and others. "Red Teaming Language Models with Language Models." In *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, 3419-3448, 2022. aclanthology.org/2022.emnlp-main.225
23. ^Phillips, P. Jonathon, Carina Hahn, Peter Fontana, Amy Yates, Kristen K. Greene, David A. Broniatowski, and Mark A. Przybocki. "Four Principles of Explainable Artificial Intelligence." NISTIR 8312, 2021. nist.gov/...es-explainable-artificial-intelligence
24. ^Liang, Percy, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, and others. "Holistic Evaluation of Language Models." *Transactions on Machine Learning Research*, 2023. openreview.net/forum
25. ^Raji, Inioluwa Deborah, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. "AI and the Everything in the Whole Wide World Benchmark." *Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks* 1, 2021. datasets-benchmarks-proceedings.neurips.cc/...t-round2
26. ^Mitchell, Margaret, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, and others. "Model Cards for Model Reporting." In *Proceedings of the Conference on Fairness, Accountability, and Transparency*, 220-229, 2019. doi.org/...3287560.3287596
27. ^Gebru, Timnit, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daume III, and Kate Crawford. "Datasheets for Datasets." *Communications of the ACM* 64, no. 12 (2021): 86-92. microsoft.com/...datasheets-for-datasets
28. ^Raji, Inioluwa Deborah, Andrew Smart, Rebecca N. White, Margaret Mitchell, Timnit Gebru, and others. "Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing." In *Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency*, 33-44, 2020. doi.org/...3351095.3372873
29. ^Brundage, Miles, Shahar Avin, Jasmine Wang, Haydn Belfield, Gretchen Krueger, and others. "Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims." arXiv, 2020. arxiv.org/...2004.07213
30. ^Autio, Chloe, Reva Schwartz, Jesse Dunietz, Shomik Jain, Martin Stanley, and others. "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile." NIST AI 600-1, 2024. nist.gov/...ork-generative-artificial-intelligence
31. ^European Commission. "AI Act." Shaping Europe's Digital Future. digital-strategy.ec.europa.eu/...ai-act
32. ^Council of Europe. "The Framework Convention on Artificial Intelligence." 2024. coe.int/...k-convention-on-artificial-intelligence