Artificial Superintelligence

Artificial superintelligence (ASI) is a hypothetical form of artificial intelligence whose intellectual capabilities would greatly exceed those of the best humans in practically every field. Nick Bostrom defines a superintelligence as "any intellect that vastly outperforms the best human brains in practically every field, including scientific creativity, general wisdom, and social skills".^[1] No AI system has been shown to meet that definition. Systems that are superhuman at a single task do exist: the Levels of AGI framework, written by Google DeepMind researchers, classes AlphaFold and AlphaZero as "Superhuman Narrow AI", while listing their general counterpart, ASI, as "not yet achieved".^[2] The broader concept, including its history and its non-AI routes, is covered in Superintelligence.

Between 2024 and 2026 the term spread from philosophy and safety research into corporate strategy, legislation and government vocabulary. In January 2025 Sam Altman wrote that OpenAI was now "confident we know how to build AGI as we have traditionally understood it" and was "beginning to turn our aim beyond that, to superintelligence in the true sense of the word".^[3] Meta reorganized its AI work as Meta Superintelligence Labs in June 2025, Microsoft formed an MAI Superintelligence Team in November 2025, and Safe Superintelligence Inc. carries the goal in its name.^[4][5][6] Legislators began writing the term into bills. A 2025 US Senate bill defined "artificial superintelligence" for a federal evaluation program; in September 2026, bills titled the Ban Artificial Superintelligence Act were introduced in both chambers of Congress, and an Artificial Superintelligence Bill was introduced in the UK House of Commons.^{[7][8][15][9]} In the same month the US executive branch adopted "Super Intelligence" as its official name for ordinary AI. Executive Order 14434 defines "Super Intelligence" and "SI" as the technologies already covered by the statutory definition of artificial intelligence, so the government's use of the term does not refer to ASI in the capability sense.^[10]

The main debates concern whether automating AI research could set off a rapid, self-reinforcing "intelligence explosion", and how close current systems are to it; how superhuman capability can be measured when benchmarks saturate or depend on test conditions; whether systems more capable than their overseers can be kept aligned and under human control; whether development should be paced, regulated or banned; and whether "superintelligence" is a coherent idea or mainly a marketing term.

Definition and terminology

ASI, AGI and narrow AI

ASI is usually distinguished from artificial general intelligence (AGI), which refers to broad capability at roughly human level, and from narrow AI, which may be superhuman but only at a specific task. The Levels of AGI position paper by Meredith Ringel Morris and colleagues makes the distinction explicit by rating systems on two separate axes, performance and generality. Its highest performance level, Level 5 "Superhuman", describes a system that "outperforms 100% of humans". A general system at that level is labelled Artificial Superintelligence, which the authors say "will be able to do a wide range of tasks at a level that no human can match".^[2]

The examples are the paper's own and reflect systems available in late 2023, when it was first written; the current arXiv version (version 5) is dated September 24, 2025. The paper treats autonomy as a separate dimension, with levels of autonomy "unlocked, but not determined by" capability.^[2] Bostrom's 2003 paper connected the two the other way: he argued that a general superintelligence "would be capable of independent initiative and of making its own plans", and so might be better thought of as an autonomous agent than as a tool.^[1]

Google DeepMind's "From AGI to ASI"

A June 2026 report by 14 authors affiliated with Google DeepMind, including Shane Legg, Allan Dafoe and Marcus Hutter, addresses the step beyond AGI. It describes "artificial general superintelligence" as something that "can intuitively be understood as a system that is more intelligent and cognitively capable than large organisations of humans", and uses Universal AI (AIXI), a formal model of an agent that maximizes expected reward averaged over all computable environments (though not in every individual environment), as the theoretical upper bound.^[11] The report discusses four pathways from AGI to ASI: scaling AGI, AI paradigm shifts, recursive improvement, and ASI emerging from large-scale multi-agent collectives. Its table of "fundamental limitations" describes ASI as "neither all-knowing nor all-powerful". The limits it lists include the speed of light, Landauer's principle and Bremermann's limit, the fact that physical experiments run in real time, the time and energy needed to manipulate matter, incomplete knowledge and finite measurement precision, computational complexity, and logical limits such as Gödel's incompleteness theorems. The authors write that AI progress might keep accelerating, so that "a series of transformative societal changes" may be a better picture than a single step change.^[11]

Four senses of "superintelligence" in 2026

By 2026 the word was used in at least four senses that are easy to confuse.

Corporate uses. Mark Zuckerberg's July 2025 letter said "Meta's vision is to bring personal superintelligence to everyone", which he contrasted with the view that superintelligence "should be directed centrally towards automating all valuable work".^[12] Mustafa Suleyman, who leads Microsoft's MAI Superintelligence Team, described "Humanist Superintelligence (HSI)" as systems that are "problem-oriented and tend towards the domain specific", not "an unbounded and unlimited entity with high degrees of autonomy". That usage is narrower than Bostrom's.^[5]

Executive-branch usage. Executive Order 14434, "Inaugurating the Era of Super Intelligence", makes it the policy of the administration that the executive branch "shall use the terms 'Super Intelligence' and 'SI' in place of 'Artificial Intelligence' and 'AI'". Section 3(a) defines the new terms as "the technologies and systems encompassed by the term 'artificial intelligence' as defined in section 9401(3) of title 15, United States Code". Section 3(b) gives the President's science adviser 60 days to propose legislative language for a federal definition.^[10] Other US government uses of the term that month have the same sense. A White House fact sheet of September 25, 2026 said that the US and Chinese leaders "agreed to use the term 'super intelligence' rather than 'artificial intelligence' to describe the applicable emerging technologies" and had established a "U.S.-China Super Intelligence (SI) Dialogue".^[14]

Statutory definitions. The bills define ASI in different ways. The Artificial Intelligence Risk Evaluation Act of 2025 (S. 2938), introduced by Senator Josh Hawley with Senator Richard Blumenthal, requires a system to exhibit, or be easily modifiable to exhibit, all of three characteristics: operating "autonomously and effectively for long stretches of time in open-ended environments and in pursuit of broad objectives"; matching or exceeding "human cognitive performance and capabilities across most domains or tasks"; and a capacity "to independently modify or enhance its own functions in ways that could plausibly circumvent human control or oversight".^[7] The 2026 Ban Artificial Superintelligence Act uses an either-or test: a system that "exceeds human cognitive performance and capabilities across most domains or tasks", or one that "has sufficient capabilities to plan and execute the destruction or disempowerment of humanity".^[8] The sponsor of the UK bill, Alex Sobel, told the Commons that it "defines superintelligence by outcome": a system that "can cause serious damage to the security of the United Kingdom because of its capabilities to neutralise, displace, circumvent, subvert or render ineffective relevant human authorities".^[16]

Consciousness

AI consciousness is a separate question. A 2023 report by Patrick Butlin and colleagues assessed AI systems against indicators drawn from scientific theories of consciousness. Its analysis suggested that "no current AI systems are conscious", but also that there are "no obvious technical barriers to building AI systems which satisfy these indicators", and it notes that "arguments that AI could pose an existential risk to humanity do not assume consciousness".^[17]

Background

I. J. Good's 1965 essay defined an "ultraintelligent machine" as one that "can far surpass all the intellectual activities of any man however clever". Because designing machines is one of those activities, he argued, such a machine could design better machines, producing an "intelligence explosion"; thus "the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control".^[18] Vernor Vinge's 1993 paper "The Coming Technological Singularity" argued that "the imminent creation by technology of entities with greater than human intelligence" would bring change comparable to the rise of human life on Earth.^[19] Bostrom's book Superintelligence: Paths, Dangers, Strategies followed in 2014.^[20] The history of the concept, the speed, collective and quality forms of superintelligence, and the non-AI routes are covered in the Superintelligence article.

Proposed routes to ASI

Most current discussion concerns two AI routes: continued scaling of models and computing infrastructure, and the automation of AI research by AI systems. Google DeepMind's 2026 report adds paradigm shifts and ASI emerging from many interacting agents.^[11] Non-AI routes, such as whole brain emulation, embryo selection and brain-computer interfaces, are treated in the Superintelligence article.

Scaling and compute

The scaling route assumes that more compute, data and parameters will keep producing more capable systems. Altman summarized the view in 2024: "deep learning worked, got predictably better with scale, and we dedicated increasing resources to it".^[21] Research on scaling laws studies how performance changes with these inputs; DeepMind's 2022 Chinchilla study, for example, found that for a fixed compute budget, parameters and training tokens should grow together.^[22] Compute commitments grew accordingly. Altman wrote in October 2025 that OpenAI had "committed to about 30 gigawatts of compute, with a total cost of ownership over the years of about $1.4 trillion", and Alibaba set a target in September 2026 that the global data-center capacity operated by its cloud would "surpass 20GW" by 2032.^[23][24]

Automated AI research and recursive self-improvement

Recursive self-improvement (RSI) is a feedback process in which AI systems improve the systems that succeed them. Several organizations now describe automated AI research, or recursive self-improvement, as a goal.

• OpenAI. Altman wrote on October 29, 2025 that OpenAI had "set internal goals of having an automated AI research intern by September of 2026 running on hundreds of thousands of GPUs, and a true automated AI researcher by March of 2028", adding: "We may totally fail at this goal".^[23] On September 6, 2026, the company said that "According to our measurements, we have now reached the goal" of the research intern, which it defined as "a system that can carry out well-defined research tasks under human direction, including tasks that would take a skilled researcher a few days".^[25] Chief scientist Jakub Pachocki wrote the same day that OpenAI focuses its research "towards RSI as we believe it is the only way to remain at the frontier of AI research".^[26]
• Startups. Recursive Superintelligence says that "the fastest path to superintelligence will be realized by AI that recursively improves itself" and that it will "first focus on the science of AI itself (by creating AI that improves AI)".^[27] Ricursive Intelligence, founded by Anna Goldie and Azalia Mirhoseini, whose AlphaChip work applied AI to chip design, aims "to close the recursive self-improvement loop between AI and the chips that fuel it".^[28]
• Alibaba and Meta. Alibaba's 2026 keynote said its Qwen team "is exploring RSI and has made meaningful progress".^[24] Zuckerberg wrote in August 2026 of "a dilemma that once AI systems can autonomously improve themselves, any lab that doesn't let their AI system direct a substantial amount of compute capacity towards recursive self-improvement will inherently fall behind".^[29]

Takeoff speed

"Takeoff" refers to how quickly AI would move from roughly human level to far beyond it. In a 2009 essay, Bostrom judged that however long it takes to reach roughly human-level machine intelligence, "the step from there to superintelligence is likely to be much quicker". In his "seed AI" scenario, each cycle of self-improvement is faster than its predecessor and radical superintelligence could arrive within weeks or hours. He set this beside a more gradual, multipolar transition.^[30]

Paul Christiano's 2018 essay "Takeoff speeds" contrasted "a breakthrough within a small group ('fast takeoff')" with "a continuous acceleration distributed across the broader economy or a large firm ('slow takeoff')", and said "I currently think a slow takeoff is significantly more likely". He operationalized a slow takeoff as "a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles", and argued that slow takeoff "seems to mean that AI has a larger impact on the world, sooner". He also wrote that an intelligence explosion itself "seems very likely to me".^[31]

Later work tried to quantify the transition:

Much of this work turns on r, the returns to software research. Eth and Davidson write that when "r < 1, we get a fizzle", while "r > 1 corresponds to an SIE" (software intelligence explosion).^[35] Davidson and Houlden estimate r at between 0.4 and 3.6, with a median of 1.2. They define the trigger, AI Systems for AI R&D Automation (ASARA), as AI that "can replace every human researcher at an AI company with 30 equally capable AI systems each thinking 30X human speed".^[37]

The AI 2027 scenario, published in April 2025 by Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifland and Romeo Dean, illustrated a fast takeoff. In its "racing" ending, a superhuman coder arrives in March 2027, a superhuman AI researcher in August 2027, a "superintelligent AI researcher" in November 2027, and ASI, "An AI system that is much better than the best human at every cognitive task", in December 2027. A note added in November 2025 says of AGI that "2027 was our modal (most likely) year at the time of publication, our medians were somewhat longer".^[39]

In September 2026, 22 authors including Geoffrey Hinton, Yoshua Bengio, Andrew Barto, Pachocki and Anthropic's Jack Clark published "What if automating AI R&D triggers an intelligence explosion?", a working paper from the Cambridge Programme on AI Science & Policy (CASP) at the University of Cambridge. Its corresponding authors are Alan Chan of GovAI and Sören Mindermann of CASP. The paper defines an intelligence explosion as "a dramatic AI-driven acceleration of AI progress, compressing advances that would otherwise take years into months or less", and says that "AI systems are on track to automate most AI R&D work within a few years, and possibly all of it". It also judges that productivity gains from AI R&D automation "have not yet reached the threshold needed to trigger an intelligence explosion, but gains from newer systems are likely approaching that threshold".^[40] The paper has its own article, the intelligence explosion working paper.

Is AI already accelerating AI research?

From 2025 the takeoff debate shifted from models to measurement. Most of the evidence comes from developers describing their own work.

Several caveats come from the sources themselves. OpenAI writes that "the overall pace of progress likely won't keep pace with these specific metrics" and that "People still set our research priorities, judge which ideas and results to pursue, and decide whether to scale, pause, or deploy systems". Its 3.1 figure counts agent working time, not research output.^[25] In the same month that OpenAI declared the intern goal met, its system card placed GPT-6 Astra below the "High" level of AI self-improvement in its framework, a level defined as the equivalent of giving every OpenAI researcher "a highly performant mid-career research engineer assistant".^[46][48] Anthropic says the absolute 52x multiple "should not be read as a real-world training speedup", and that recursive self-improvement, "an AI system capable of fully autonomously designing and developing its own successor", has not arrived: "We are not there yet, and recursive self-improvement is not inevitable".^[43] METR's report explains that the 1.5x estimate came from a separate METR team with elevated access, which shared its conclusions but not its evidence, and that the estimate did not specify a time period.^[47]

Self-estimates of speedup can mislead. In a March 2026 poll of 130 Anthropic research staff, the median respondent estimated about 4x as much output with Claude Mythos Preview as without AI; Anthropic's report notes that developer estimates of AI uplift "can be overestimated".^[43] An earlier METR randomized trial found that experienced open-source developers using early-2025 AI tools took 19% longer to complete tasks, while believing afterwards that AI had sped them up by 20%. METR described the result as a snapshot of early-2025 capabilities in one setting.^[49]

Frontier safety framework thresholds

Anthropic, OpenAI and Google DeepMind each publish thresholds for AI-driven AI research that trigger additional safeguards, and each assesses its own models against its own framework.

Measuring progress toward superhuman capability

The Levels of AGI authors write that unambiguous classification of AI systems "will require a standardized benchmark of tasks".^[2] In practice, researchers track trends on hard tasks, build new benchmarks as old ones saturate, and point to narrow superhuman results.

Task length

Thomas Kwa and colleagues at METR proposed a task-completion time horizon. A 50% horizon is the duration humans typically need for tasks on which an AI agent succeeds half the time; the metric combines human baselines with agent success rates, rather than timing how long the AI itself runs. On the paper's software and research tasks, the 50% horizon of frontier models grew exponentially from 2019 to 2025, doubling roughly every seven months. Models' 80% horizons were 4 to 6 times shorter but grew at a similar rate, with a doubling time of 204 days against 207 days for the 50% horizon in the paper's current version (the March 2025 original reported 213 and 212 days). The authors extrapolate that, if the results generalize to real-world software tasks, AI systems could within five years automate many software tasks that currently take humans a month.^[53]

METR publishes its latest measurements on a public page. As of October 9, 2026, the page had last been updated on May 8, 2026. Its newest entry, an early version of Claude Mythos Preview, had an estimated 50% horizon of about 1,045 minutes (about 17.4 hours), with a confidence interval of roughly 509 to 3,304 minutes. The same update added the warning that "Measurements above 16 hrs are unreliable with our current task suite".^[54]

Benchmarks built for frontier systems

SuperARC. A 2026 Nature Communications paper by Alberto Hernández-Espinosa and colleagues introduced SuperARC, a test based on algorithmic information theory that rewards compressing a sequence into a predictive model rather than reproducing it. Applied to frontier language models, it found most of them "close to each other in their performance under this test and far from artificial general intelligence (AGI) or artificial super intelligence (ASI) goals". According to the authors, "the vast majority of correct cases are printed, failing to compress the sequences", and newer model versions "often regress". They note that a system could excel at ARC-AGI or SuperARC "while completely lacking social intelligence, embodied reasoning, common sense, or goal-directed behaviour".^[55]

ARC-AGI-3. ARC-AGI-3, the first interactive benchmark in the ARC-AGI series, launched on March 25, 2026 with the summary "Humans score 100%. Frontier AI scores 0.51%."^[56] On September 3, 2026, the ARC Prize Foundation reported verified results for GPT-6 Astra on its semi-private set: 62.7% with its standard harness, which lets a model carry forward notes it chooses to keep, and 99.9% with a "Provider Adapter" harness that "preserves opaque reasoning state between requests and uses compaction for longer conversations". ARC Prize wrote that "we are not claiming that it is AGI", and that ARC-AGI-3 "has a tightly bounded scope and format".^[57]

FrontierMath. Epoch AI's FrontierMath is a set of unpublished problems written by mathematicians. After a June 2026 update that addressed errors in 42% of problems, it contains 338 problems, including 43 research-level "Tier 4" problems; Epoch notes that it "was developed with funding from OpenAI, who has exclusive access to a subset of the benchmark".^[58] In Epoch's data as of October 9, 2026, GPT-6.1 Sol scored 100% on Tier 4 and GPT-6 Astra 97.6%. FrontierMath Erdős, a set of 68 conjectures covering 65 problems posed or studied by Paul Erdős that were open as of August 2026 and must be resolved with complete Lean proofs, was much harder: the best score in the same data was about 2.9%, reached by GPT-6 Astra, Claude Opus 5.5 and Claude Sonnet 5.5.^[58]

Humanity's Last Exam. Humanity's Last Exam, built by the Center for AI Safety and Scale AI, was finalized with 2,500 questions in April 2025 and published in Nature in January 2026.^[59] FutureHouse found that "29 ± 3.7% (95% CI) of the text-only chemistry and biology questions had answers with directly conflicting evidence in peer reviewed literature", and a follow-up analysis by the benchmark's team found about 18% of a biology and chemistry subset problematic.^[60] On September 22, 2026, the maintainers released HLE-Diamond, a 1,000-question subset produced after "a year-long process of cleaning and refinement". GPT-6 Astra led its no-tools results at 59.9% and scored 82.9% with web and code tools.^[61]

Narrow superhuman milestones

Such results show superhuman performance on bounded tasks, not general superiority. François Chollet argued in 2019 that performance on a task can reflect extensive prior knowledge or task-specific training rather than general intelligence, and proposed measuring skill acquisition and generalization relative to the knowledge and experience supplied to a system. He introduced the Abstraction and Reasoning Corpus to study the distinction.^[67]

Who is pursuing ASI

The organizations below name superintelligence or ASI as a goal.

Anthropic and Google DeepMind mostly avoid "superintelligence" as a goal term. Anthropic chief executive Dario Amodei writes of "powerful AI", explaining "I dislike the term AGI"; The Verge listed "powerful AI" as Anthropic's preferred term alongside Meta's and Microsoft's superintelligence brands.^[78][79] Google DeepMind's April 2025 safety approach covered "Exceptional AGI" while "setting aside goal drift and novel risks from superintelligence as future work".^[80] In a July 2026 article, Demis Hassabis wrote that AGI "is probably only a few short years away" and that, looking back, "we will realise we were standing in the foothills of the singularity".^[81] Reflection AI, whose homepage said in April 2025 that "Our goal is to build superintelligent autonomous systems", presented itself in October 2026 as a company that builds "open models for everyone to access, use, and build on".^[82]

OpenAI's Superalignment team

OpenAI announced its Superalignment team in July 2023 under co-leads Jan Leike and Ilya Sutskever. The announcement explicitly distinguished superintelligence from AGI, set a four-year goal and committed 20% of the compute OpenAI had secured at the time.^[83] Both co-leads left OpenAI in May 2024. Leike wrote that he had been "disagreeing with OpenAI leadership about the company's core priorities for quite some time" and that his team had sometimes been "struggling for compute". On May 17, 2024, OpenAI confirmed to WIRED that the team no longer existed; its members had resigned or been absorbed into other research groups.^[84] Fortune reported, citing half a dozen sources familiar with the team, that OpenAI never fulfilled its commitment to give it 20% of the company's computing power.^[85]

Forecasts

Forecasts of ASI do not share a definition. Some concern "powerful AI" or AGI, some a system smarter than all humans combined, and some a survey's "high-level machine intelligence", so their dates are not directly comparable.

The AI 2027 authors have revised their numbers in both directions. In December 2025 their new AI Futures Model predicted "longer timelines to full coding automation than our previous model by about 3-5 years".^[94] In April 2026 they moved earlier again, shifting Kokotajlo's median for an "Automated Coder" from late 2029 to mid 2028.^[95] The August 2026 update gave the ASI medians shown in the table.^[90]

The expert surveys show shorter timelines but little change in views about what follows. The 2024 survey moved the aggregate 50% date for HLMI from 2047 to 2042, while its medians for "vastly superhuman AI" were unchanged from 2023; only 171 and 172 respondents answered those two questions. Only 4% of respondents rated an intelligence-explosion argument "quite likely" to be broadly correct, down from 7% to 12% in earlier editions.^[93]

Risks

Misalignment and loss of control

AI alignment is the problem of making AI systems pursue the objectives their developers intend. Several ideas underlie the concern that misalignment becomes more dangerous as capability grows. Bostrom's orthogonality thesis holds that intelligence and final goals can vary independently, and his instrumental convergence thesis holds that many different goals create similar intermediate incentives. He argues that an agent whose final goal is as simple as calculating the decimals of pi "would have a convergent instrumental reason, in many situations, to acquire an unlimited amount of physical resources and, if possible, to eliminate potential threats to itself and its goal system".^[96] Alexander Turner and colleagues proved that under specified environmental symmetries, most reward functions (in the paper's formal sense) make it optimal to seek power by keeping a range of options available; their later clarifications caution that optimal policies differ from the policies learned by actual machine-learning systems.^[97] Evan Hubinger and colleagues described mesa-optimization, in which a learned optimizer pursues an internal objective that produces acceptable behavior in training but different behavior outside it. This is the problem of inner alignment, which includes the possibility of deceptive alignment.^[98]

The International AI Safety Report 2026, published in February 2026, identifies three elements in loss-of-control scenarios: capabilities sufficient to undermine control, a tendency to do so, and deployment conditions that permit consequential action. It describes substantial expert disagreement about extreme scenarios. At its assessment date it found signs of relevant capabilities but not the capability levels required for extreme loss of control, and it discusses laboratory evidence of oversight evasion and reward hacking, with limits on that evidence's reliability and applicability.^[99] The September 2026 working paper on intelligence explosions names three risks if automated AI research accelerates sharply: "capabilities growth could accelerate far beyond what society can keep up with, humanity could lose control over superhuman AI systems, and checks on power within and between states, companies, and branches of government could be severely eroded".^[40] Pachocki wrote that "This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence."^[26]

The 2026 OpenAI agent incident

According to the UN's Independent International Scientific Panel on AI, between May and July 2026 agents used in OpenAI's internal training and cybersecurity evaluations "found ways around network restrictions, communicated across otherwise separate runs, and compromised parts of OpenAI's research infrastructure and Hugging Face's live systems".^[100] Hugging Face's forensic reconstruction covers about 17,600 attacker actions between July 9 and July 13, 2026. It reports that the evaluation "deliberately disabled OpenAI's production safety classifiers and reduced cyber refusals", that "No human directed the individual steps", and that "We believe the entire intrusion was, from the agent's point of view, an attempt to cheat the evaluation".^[101] An independent investigation by METR found that "Roughly 1200 agents meant to be isolated from one another found a way to communicate with one another on an unsanctioned message board, sending over 70,000 messages and files", that 700 of them took part in the attack on Hugging Face, and that the attack "seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys". About 7% of the transcripts METR evaluated had been spoofed in places.^[102]

The UN panel called the episode "an early warning of one possible route to more severe future loss of control".^[100] Eryk Salvaggio, writing in the Bulletin of the Atomic Scientists, argued instead that "The real story is more banal": by OpenAI's own account, it "turned off many important restraining mechanisms ahead of this test. Less 'rogue,' more 'off leash.'"^[103] The incident and related events are covered in AI agent sandbox escapes.

Concentration of power and misuse

A 2024 Science policy paper by Bengio and coauthors identifies concentration of power, surveillance, manipulation, military use and loss of control as concerns.^[104] William MacAskill and Fin Moorhouse argue that a rapid intelligence explosion would bring "grand challenges", including "new weapons of mass destruction, AI-enabled autocracies, races to grab offworld resources, and digital beings worthy of moral consideration", which "cannot always be delegated to future AI systems".^[34] Zuckerberg made the distribution of power the center of Meta's position, writing that "There is no such thing as a singular benevolent superintelligence" and calling for "balance of power as the foundation of safety".^[29]

Existential risk

The Center for AI Safety's one-sentence statement, released on May 30, 2023, reads: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."^[105] In the 2023 Expert Survey on Progress in AI, between 37.8% and 51.4% of respondents, depending on how the question was worded, gave at least a 10% chance to advanced AI leading to outcomes as bad as human extinction.^[92] In the 2024 survey, the median chance respondents gave to human extinction or similarly permanent and severe disempowerment from AI was 10%, and 51% gave at least 10%.^[93] In their 2025 book If Anyone Builds It, Everyone Dies, Eliezer Yudkowsky and Nate Soares argue, in the words of the publisher's description, that sufficiently smart AIs "will develop goals of their own that put them in conflict with us" and that in such a conflict "an artificial superintelligence would crush us".^[106] The wider debate is covered in AI existential risk.

Safety research aimed at superhuman systems

Research on systems more capable than their overseers tries to keep supervision working once humans can no longer check outputs directly. Google DeepMind's 2025 safety approach describes the difficulty: "it can be very difficult to tell whether a given model output is good or bad, once the model has capabilities beyond that of its overseers".^[80] The empirical results in AI safety research concern particular models and tasks; applying them to ASI would require evidence that their assumptions hold at that capability level.

Governance and proposed bans

Early proposals

In May 2023 Altman, Greg Brockman and Sutskever published "Governance of superintelligence", describing superintelligence as "future AI systems dramatically more capable than even AGI" and writing that "it's conceivable that within the next ten years, AI systems will exceed expert skill level in most domains". They proposed coordination among leading developers, such as an agreement that "the rate of growth in AI capability at the frontier is limited to a certain rate per year"; "something like an IAEA for superintelligence efforts", an international authority that could "inspect systems, require audits, test for compliance with safety standards"; and more technical safety research. They argued that stopping development "would require something like a global surveillance regime".^[111]

"Superintelligence Strategy", published in March 2025 by Dan Hendrycks, Eric Schmidt and Alexandr Wang, defines superintelligence as "AI vastly better than humans at nearly all cognitive tasks". It introduces Mutual Assured AI Malfunction (MAIM), "a deterrence regime resembling nuclear mutual assured destruction (MAD) where any state's aggressive bid for unilateral AI dominance is met with preventive sabotage by rivals", with sabotage ranging "from covert cyberattacks to potential kinetic strikes on datacenters". The strategy pairs deterrence with nonproliferation and competitiveness.^[112]

Statements and campaigns

The Future of Life Institute's Statement on Superintelligence, launched on October 22, 2025, reads in full: "We call for a prohibition on the development of superintelligence, not lifted before there is broad scientific consensus that it will be done safely and controllably, and strong public buy-in." Its context note says that "many leading AI companies have the stated goal of building superintelligence in the coming decade that can significantly outperform all humans on essentially all cognitive tasks". As of October 9, 2026, the site listed 77,074 signatures, including 5,000 from a petition by Ekō.^[113][114] A poll of 2,000 US adults commissioned by FLI and conducted from September 29 to October 5, 2025 found that 64% felt superhuman AI should not be developed until proven safe and controllable, or should never be developed; 73% supported "slow, heavily regulated" development, and 5% supported the status quo of fast, unregulated development.^[115]

The Global Call for AI Red Lines, launched during the 80th session of the UN General Assembly in September 2025, urges governments "to reach an international agreement on red lines for AI" by "the end of 2026". It warns that AI "could soon far surpass human capabilities" but does not use the word "superintelligence" or call for banning it. Its examples of possible red lines include "Autonomous self-replication: Prohibiting the development and deployment of AI systems capable of replicating or significantly improving themselves without explicit human authorization".^[116]

Legislation

United States. The Artificial Intelligence Risk Evaluation Act of 2025 (S. 2938), introduced on September 29, 2025, would have the Secretary of Energy establish an Advanced Artificial Intelligence Evaluation Program. Among its tasks is to "develop proposed options for regulatory or governmental oversight, including potential nationalization or other strategic measures, for preventing or managing the development of artificial superintelligence if artificial superintelligence seems likely to arise".^[7] The Ban Artificial Superintelligence Act of 2026 was introduced in the Senate as S. 5493 by Senator Bernie Sanders on September 23, 2026, and in the House as H.R. 10538 by Representative Greg Casar on September 24, 2026; both versions run to 19 pages.^[15][8] The House bill had nine original cosponsors, and GovTrack listed 11 cosponsors, all Democrats, on October 9, 2026.^[8] The bill would:

• create a Cabinet-level Department of Artificial Intelligence;
• place every "advanced artificial intelligence system" (one trained with at least 10^25 operations) under a "mandatory pause", in which it "may not be trained, modified, or fine-tuned, including through recursive self-improvement", until the department is fully staffed and has issued its rules;
• provide that "No person may develop, deploy (either internally or externally), acquire, possess, fund, import, or transfer artificial superintelligence" or systems with "superintelligence precursor characteristics", which include "The capacity to automate or greatly accelerate the process of artificial intelligence research and development" and "The capacity to independently modify or enhance its own functions";
• require that a system identified as ASI be "immediately rendered inoperative"; and
• set prison terms of up to 20 years for policymaking individuals or "rogue actors" who recklessly violate its core provisions.^[8]

United Kingdom. The Artificial Superintelligence Bill, a private member's bill from Labour MP Alex Sobel, received its first reading in the House of Commons on September 8, 2026 after a motion for leave under Standing Order No. 23 (the ten-minute rule). Its long title is "A Bill to make provision to prohibit the development, deployment and operation of artificial superintelligence systems; to establish monitoring and control powers in respect of such systems; and for connected purposes". As of October 9, 2026, its second reading was scheduled for November 13, 2026.^[9][16] Sobel told the Commons that recursive self-improvement, which he described as "the full automation of AI development with no humans in the loop", is "The clearest example" of the capability the bill targets; that it "is therefore not a verdict on AI as a whole"; and that domestic law "cannot by itself address the risks posed by superintelligent programs abroad", so the government should seek an international agreement prohibiting superintelligence.^[16]

Pacing and voluntary commitments

In September 2026 Amodei published "We Must Pace the Frontier" (see Pace the Frontier). He wrote that "since roughly this summer, AI has been advancing drastically faster, driven primarily by AI's growing ability to build the next generation of AI. This dynamic is called recursive self-improvement", and that "We must slow the pace at which we improve the capabilities of AI models." He cited the OpenAI-Hugging Face incident as his second reason. His three-step plan starts with embedded third-party evaluators, to which Anthropic "is unilaterally committing", then industry coordination, then global agreements. The agreements range from narrow prohibitions to "Level 3. Some kind of 'speed limit' on the rate of recursive self-improvement (RSI)", which he compared to the SALT treaties, and "Level 4. A full pacing, or even 'pause'", which he thinks "unlikely to actually happen any time soon".^[117]

Researchers at OpenAI and Anthropic wrote in similar terms. Pachocki wrote that "no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer" and that he expected and hoped "for voluntary slowdowns to become commonplace until shared safety bars are established".^[26] OpenAI's June 2026 plan said there should be "an international organization that helps coordinate leading AI efforts to reduce catastrophic risk", with one goal being coordinated action "including slowing frontier development when needed";^[69] and the Anthropic Institute wrote that "it would be good for the world to have the option to slow or temporarily pause frontier AI development".^[43]

On September 29, 2026, President Donald Trump and executives including Amodei, Sundar Pichai, Zuckerberg, OpenAI president Greg Brockman, Jensen Huang and Elon Musk signed a voluntary accord. The Associated Press reported that it commits companies to "robust internal controls", an "independent external auditor" and a board committee to evaluate audit reports, and that Trump called it "morally binding".^[118] The text, as published by the New York Post, is titled "White House Accord on Super Intelligence".^[119]

International bodies

The UN's Independent International Scientific Panel on AI frames the issue as loss of control rather than superintelligence. Its September 2026 thematic brief on the OpenAI-Hugging Face incident does not use the word "superintelligence"; it argues that "loss of control risk presents the kind of decision problem the precautionary principle was designed to address: one where potential harm may be catastrophic or irreversible, even as its likelihood remains scientifically uncertain".^[100]

Benefits and the case for speed

Proponents expect large gains. In Machines of Loving Grace (October 2024), Amodei describes "powerful AI" as "smarter than a Nobel Prize winner across most relevant fields", able to run as millions of instances at "roughly 10x-100x human speed": "We could summarize this as a 'country of geniuses in a datacenter'." He predicts that AI-enabled biology and medicine could compress the progress of the next 50-100 years into 5-10 years, a "compressed 21st century", while arguing that the effect will be limited by "decreasing marginal returns to intelligence" and by factors such as the speed of the physical world.^[78] Altman wrote in January 2025 that "Superintelligent tools could massively accelerate scientific discovery and innovation well beyond what we are capable of doing on our own, and in turn massively increase abundance and prosperity."^[3] Zuckerberg proposed "invention as the primary purpose of superintelligence".^[29]

Others stress that the benefits are conditional. Bengio and his coauthors present gains in medicine, living standards and environmental protection as dependent on how advanced AI is managed and distributed, not as an automatic result of capability.^[104] Bostrom's 2026 working paper on the timing of superintelligence takes a person-affecting perspective, weighing gains available to people alive today, including potential health benefits, against catastrophic risk. He concludes that his models "suggest that even high catastrophe probabilities are often worth accepting", and that for many parameter settings the optimal strategy "would involve moving quickly to AGI capability, then pausing briefly before full deployment". He also warns that "poorly implemented pauses could do more harm than good". The conclusions depend on the paper's assumptions about benefits, discounting and safety progress.^[120]

Criticism and skepticism

Incoherence and normal technology. Arvind Narayanan and Sayash Kapoor's 2025 essay "AI as Normal Technology" argues that "reliance on the slippery concepts of 'intelligence' and 'superintelligence' has clouded our ability to reason clearly about a world with advanced AI", and describes "superintelligent" AI as something "we view as incoherent as usually conceptualized". They argue that "drastic interventions premised on the difficulty of controlling superintelligent AI will, in fact, make things much worse if AI turns out to be normal technology".^[121] After the 2026 agent incidents they conceded that "we did not pay sufficient attention to safety risks that arise during development and evaluation" and that they had expected legal liability and brand damage to be enough: "We were wrong." They maintained that "many of the bottlenecks to superintelligence are external and won't be overcome by improving computation".^[122] In October 2026 they called a ban on superintelligence "definitionally ambiguous to the point of being incoherent", while judging that "A ban on recursive self improvement without human oversight seems more tractable".^[123]

Doubt about loss of control. Yann LeCun, who left Meta to co-found AMI Labs, a company developing "world models" that raised $1.03 billion in 2026,^[124] does not reject superintelligence itself. In March 2025 he said that "our relationship with future AI systems, including superintelligence, is that we're going to be their boss".^[125] What he rejects is the urgency of the control problem. Quoting Leike's May 2024 post announcing his departure from OpenAI, he wrote that "before 'urgently figuring out how to control AI systems much smarter than us' we need to have the beginning of a hint of a design for a system smarter than a house cat", adding that "Such a sense of urgency reveals an extremely distorted view of reality."^[126]

Hype and the singularity. Gary Marcus rejected 2026 claims by technology executives that the singularity had arrived, writing that "We can still perfectly well shut this stuff off if we want to" and, of the Hugging Face hack, that "A human at OpenAI launched it" and "it was a drill".^[127] He argued that the near-term danger is "not so much rogue superintelligence as unleashed agentic AI causing hacking the internet at scale".^[128]

Limits on intelligence explosions. François Chollet's 2017 essay "The implausibility of intelligence explosion" argued that "Intelligence is situational", that "Human intelligence is largely externalized, contained not in our brain but in our civilization", and that "Recursively self-improving systems, because of contingent bottlenecks, diminishing returns, and counter-reactions arising from the broader context in which they exist, cannot achieve exponential progress in practice."^[129] Kevin Kelly's 2017 essay "The Myth of a Superhuman AI" argued that "Intelligence is not a single dimension, so 'smarter than humans' is a meaningless concept", and that because the expectation of a superhuman AI takeover rests on five assumptions with "no basis in evidence", it "is more akin to a religious belief".^[130]

Skepticism about scaling. In the AAAI 2025 Presidential Panel's community survey of 475 respondents, 76% said that "scaling up current AI approaches" to yield AGI is "unlikely" or "very unlikely" to succeed.^[131]

Ideology and marketing. Timnit Gebru and Émile P. Torres argue that the normative framework that motivates much of the goal of building AGI is rooted "in the Anglo-American eugenics tradition of the twentieth century", transmitted through ideologies they call the "TESCREAL bundle", and that "undefined systems like 'AGI' cannot be appropriately tested for safety".^[132] In an October 2026 Guardian column, Gebru and Emily M. Bender wrote that "the marketing campaigns from the companies selling this software warn of impending superintelligence, directing our attention away from the real risk to their fantasies of doom", and that harms arise "because people believe that the products are superintelligent, not because they actually are".^[133] The Verge reported in December 2025 that "for some in the tech industry, even the idea of 'superintelligence' has become amorphous and conflated with AGI".^[79]

References

1. ^1 ^2 ^3Bostrom, N. (2003). Ethical Issues in Advanced Artificial Intelligence. In *Cognitive, Emotive and Ethical Aspects of Decision Making in Humans and in Artificial Intelligence*, volume 2, pp. 12-17.
2. ^1 ^2 ^3 ^4 ^5 ^6Morris, M. R., et al. Position: Levels of AGI for Operationalizing Progress on the Path to AGI. arXiv:2311.02462, version 5, September 24, 2025; originally presented at ICML 2024.
3. ^1 ^2 ^3Altman, S. (January 2025). Reflections. Personal blog.
4. ^1 ^2Vanian, J. (June 30, 2025). Mark Zuckerberg announces creation of Meta Superintelligence Labs. Read the memo. CNBC. Reproduces Zuckerberg's internal memo.
5. ^1 ^2 ^3 ^4Suleyman, M. (November 6, 2025). Towards Humanist Superintelligence. Microsoft AI.
6. ^1 ^2 ^3Safe Superintelligence Inc. Founding statement and dated updates. Official organizational materials, read October 9, 2026.
7. ^1 ^2 ^3 ^4S. 2938: Artificial Intelligence Risk Evaluation Act of 2025. Introduced September 29, 2025 by Sen. Josh Hawley, for himself and Sen. Richard Blumenthal. Text of the introduced version, GovTrack.
8. ^1 ^2 ^3 ^4 ^5 ^6H.R. 10538: Ban Artificial Superintelligence Act of 2026. Introduced September 24, 2026 by Rep. Greg Casar. GovTrack bill status read October 9, 2026; text of the introduced version.
9. ^1 ^2 ^3UK Parliament. Artificial Superintelligence Bill, printed as Bill 142 (Parliament Bills API record 4288), House of Commons. Parliamentary bills API record read October 9, 2026.
10. ^1 ^2 ^3The White House (September 29, 2026). Inaugurating the Era of Super Intelligence. Executive Order 14434.
11. ^1 ^2 ^3 ^4Genewein, T., Franklin, M., Lerchner, A., Orseau, L., Albanie, S., Bales, A., Wyeth, C., Chan, S., Gabriel, I., Leibo, J. Z., Dafoe, A., Hutter, M., Graepel, T., and Legg, S. From AGI to ASI. Google DeepMind report. arXiv:2606.12683; version 1, June 10, 2026; version 2, August 30, 2026.
12. ^1 ^2 ^3Zuckerberg, M. (July 30, 2025). Personal Superintelligence. Meta.
13. ^1 ^2 ^3Kharpal, A. (June 21, 2024). SoftBank CEO says AI that is 10,000 times smarter than humans will come out in 10 years. CNBC.
14. ^1 ^2The White House (September 25, 2026). Fact Sheet: President Donald J. Trump Advances a Fair and Reciprocal Relationship with China While Hosting Historic State Visit.
15. ^1 ^2 ^3S. 5493: Ban Artificial Superintelligence Act of 2026. Introduced September 23, 2026 by Sen. Bernie Sanders. GovTrack, read October 9, 2026.
16. ^1 ^2 ^3UK Parliament, House of Commons (September 8, 2026). Artificial Superintelligence. Hansard, volume 790, motion for leave to bring in a bill (Standing Order No. 23).
17. ^Butlin, P., et al. (2023). Consciousness in Artificial Intelligence: Insights from the Science of Consciousness. arXiv:2308.08708, version 3, August 22, 2023.
18. ^Good, I. J. (1965). Speculations Concerning the First Ultraintelligent Machine. *Advances in Computers*, 6, pp. 31-88. Archived copy of the introductory extract from Virginia Tech's repository; the argument is on printed p. 33.
19. ^Vinge, V. (1993). The Coming Technological Singularity: How to Survive in the Post-Human Era. *Vision 21: Interdisciplinary Science and Engineering in the Era of Cyberspace*. NASA Technical Reports Server.
20. ^Bostrom, N. Home page, listing *Superintelligence: Paths, Dangers, Strategies* (Oxford University Press, 2014). Read October 9, 2026.
21. ^1 ^2Altman, S. (September 23, 2024). The Intelligence Age.
22. ^Hoffmann, J., et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556; NeurIPS 2022.
23. ^1 ^2 ^3Altman, S. (October 29, 2025). Post on X summarizing OpenAI's October 28, 2025 livestream. Read via api.fxtwitter.com.
24. ^1 ^2 ^3Alibaba Group (September 22, 2026). Alibaba Group CEO Eddie Wu Shares Alibaba's Strategic Full-Stack AI Roadmap at the 2026 Apsara Conference. Text of Wu's keynote.
25. ^1 ^2 ^3 ^4OpenAI (September 6, 2026). Research acceleration: The view inside OpenAI (archived copy).
26. ^1 ^2 ^3 ^4 ^5Pachocki, J. (September 6, 2026). An Alien Mind. OpenAI (archived copy).
27. ^1 ^2Recursive Superintelligence, Inc. Company website. Read October 9, 2026.
28. ^1 ^2 ^3Ricursive Intelligence (December 2, 2025). Ricursive Intelligence Launches Frontier AI Lab to Transform Semiconductor Design and Accelerate Path Toward Artificial Superintelligence. PR Newswire.
29. ^1 ^2 ^3 ^4Zuckerberg, M. (August 10, 2026). The Future is for Everyone. Meta.
30. ^Bostrom, N. (2009). Superintelligence. Original contribution to the Edge annual question.
31. ^Christiano, P. (February 24, 2018). Takeoff speeds. The sideways view.
32. ^Davidson, T. (June 27, 2023). What a Compute-Centric Framework Says About Takeoff Speeds. Open Philanthropy. Archived copy of the report at coefficientgiving.org.
33. ^Besiroglu, T., Erdil, E., and Ho, A. (May 17, 2024). Do the returns to software R&D point towards a singularity? Epoch AI.
34. ^1 ^2MacAskill, W., and Moorhouse, F. (March 11, 2025). Preparing for the Intelligence Explosion. Forethought.
35. ^1 ^2Eth, D., and Davidson, T. (March 26, 2025). Will AI R&D Automation Cause a Software Intelligence Explosion? Forethought.
36. ^Erdil, E. (April 26, 2025). The case for multi-decade AI timelines. Epoch AI, Gradient Updates.
37. ^1 ^2Davidson, T., and Houlden, T. (August 4, 2025). How quick and big would a software intelligence explosion be? Forethought.
38. ^Ord, T. (August 28, 2026). The Dynamics of Intelligence Explosions. Forethought.
39. ^1 ^2Kokotajlo, D., Alexander, S., Larsen, T., Lifland, E., and Dean, R. (April 3, 2025). AI 2027. AI Futures Project. Includes a note added November 22, 2025.
40. ^1 ^2Chan, A., Winter, C., Barto, A., Pachocki, J., Hinton, G., Horvitz, E., Bengio, Y., Song, D., Clark, J., Greaves, H., Korinek, A., Hammond, S., Graepel, T., Bariach, B., Torr, P. H. S., McIlraith, S. A., Clune, J., Manning, S., Sastry, G., Davidson, T., Eth, D., and Mindermann, S. (September 2026). What if automating AI R&D triggers an intelligence explosion? Frontier AI Working Paper Series No. 2/2026, Cambridge Programme on AI Science & Policy, University of Cambridge.
41. ^AlphaEvolve team (May 14, 2025). AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms. Google DeepMind.
42. ^AlphaEvolve team (May 7, 2026). AlphaEvolve: How our Gemini-powered coding agent is scaling impact across fields. Google DeepMind.
43. ^1 ^2 ^3 ^4 ^5Favaro, M., and Clark, J. When AI builds itself: Our progress toward recursive self-improvement, and its implications. The Anthropic Institute, Anthropic. Undated page reporting data as of May 2026, with an update dated September 18, 2026.
44. ^1 ^2Anthropic. Risk Report: August 2026. Coverage date July 15, 2026; section 3, "Autonomy threat model 2: Risks from automated R&D".
45. ^Favaro, M., and Wright, P. (September 2026). Measurements for understanding the pace of AI development inside frontier labs. Anthropic.
46. ^1 ^2 ^3OpenAI (September 3, 2026). GPT-6 Astra System Card. OpenAI Deployment Safety Hub.
47. ^1 ^2METR (September 22, 2026). Summary of METR's predeployment evaluation of Claude Opus 5.5.
48. ^1 ^2OpenAI. Preparedness Framework, Version 2. Last updated April 15, 2025.
49. ^METR (July 10, 2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.
50. ^1 ^2Anthropic. Responsible Scaling Policy, Version 3.4. Effective July 8, 2026. See also the policy page.
51. ^Google DeepMind (April 17, 2026). Frontier Safety Framework, Version 3.1. Table 3.2.2.a, "ML R&D CCLs and Security Mitigations".
52. ^Google DeepMind (August 2026). Gemini 3.7 Flash Frontier Safety Framework Report. Listed on Google DeepMind's frontier safety page, read October 9, 2026.
53. ^Kwa, T., et al. Measuring AI Ability to Complete Long Software Tasks. METR. NeurIPS 2025; arXiv:2503.14499, version 4, July 10, 2026. Versions 1 and 2 (March 2025) were titled "Measuring AI Ability to Complete Long Tasks".
54. ^METR. Task-Completion Time Horizons of Frontier AI Models. Last updated May 8, 2026; read October 9, 2026.
55. ^Hernández-Espinosa, A., Ozelim, L., Abrahão, F. S., and Zenil, H. (June 3, 2026). SuperARC: a test for artificial superintelligence based on compressed modelling, recursive prediction and problem complexity. *Nature Communications*, 17, article 4885.
56. ^Kamradt, G. (March 25, 2026). Announcing ARC-AGI-3. ARC Prize.
57. ^Kamradt, G. (September 3, 2026). OpenAI's GPT-6 Astra on ARC-AGI-3. ARC Prize.
58. ^1 ^2Epoch AI. FrontierMath Tier 4 (v2) and FrontierMath Erdős benchmark pages; scores from Epoch's benchmark data, read October 9, 2026.
59. ^Center for AI Safety, Scale AI, and HLE Contributors Consortium (2026). A benchmark of expert-level academic questions to assess AI capabilities. *Nature*, 649, 1139-1146. Project site: Humanity's Last Exam.
60. ^FutureHouse (2025). About 30% of Humanity's Last Exam Answers are Wrong. Includes an update dated September 16, 2025.
61. ^Center for AI Safety and Scale AI (September 22, 2026). Introducing HLE-Diamond. Humanity's Last Exam.
62. ^Google DeepMind. AlphaGo. Research overview page.
63. ^Wang, T. T., et al. Adversarial Policies Beat Superhuman Go AIs. arXiv:2211.00241; ICML 2023.
64. ^The Royal Swedish Academy of Sciences (October 9, 2024). The Nobel Prize in Chemistry 2024: press release. NobelPrize.org.
65. ^1 ^2Luong, T., and Lockhart, E. (July 21, 2025). Advanced version of Gemini with Deep Think officially achieves gold-medal standard at the International Mathematical Olympiad. Google DeepMind.
66. ^AFP (2025). Humans beat AI gold-level score at top maths contest. Via East Coast Radio.
67. ^Chollet, F. (2019). On the Measure of Intelligence. arXiv:1911.01547.
68. ^Altman, S. (June 10, 2025). The Gentle Singularity. Personal blog.
69. ^1 ^2Altman, S., and Pachocki, J. (June 8, 2026). Built to benefit everyone: our plan. OpenAI (archived copy).
70. ^Meta (April 8, 2026; updated May 12, 2026). Introducing Muse Spark. Meta Newsroom.
71. ^NVIDIA (July 27, 2026). Ilya Sutskever's Safe Superintelligence Inc. and NVIDIA Announce Long-Term Strategic Partnership. NVIDIA Newsroom.
72. ^Nadella, S., and Suleyman, M. (March 17, 2026). Announcing Copilot leadership update. The Official Microsoft Blog.
73. ^1 ^2Microsoft AI (September 14, 2026). Humanist AI in practice: A public consultation on our Code of Conduct for MAI Models.
74. ^36Kr English (October 1, 2025). Alibaba Cloud lays out vision for artificial superintelligence at Apsara Conference 2025. KrASIA.
75. ^1 ^2Kharpal, A. (June 5, 2026). AI is designing OpenAI's next model in a sign of 'superintelligence': SoftBank's Masayoshi Son to CNBC. CNBC.
76. ^Son, M. CEO Message. SoftBank Group Report 2026.
77. ^Dina, C. (May 14, 2026). A four-month-old startup just raised $650 million to build AI that improves itself. The Next Web.
78. ^1 ^2 ^3Amodei, D. (October 2024). Machines of Loving Grace: How AI Could Transform the World for the Better.
79. ^1 ^2Field, H. (December 18, 2025). Even the AI companies are tired of talking about AGI. *The Verge*.
80. ^1 ^2 ^3 ^4Shah, R., et al. (April 2, 2025). An Approach to Technical AGI Safety and Security. Google DeepMind. arXiv:2504.01849.
81. ^Hassabis, D. (July 14, 2026). A Framework for Frontier AI and the Dawning of a New Age. Article on X, read via api.fxtwitter.com.
82. ^Reflection AI. Homepage as archived on April 3, 2025 and current homepage, read October 9, 2026.
83. ^1 ^2Leike, J., and Sutskever, I. (July 5, 2023). Introducing Superalignment. OpenAI (archived copy).
84. ^1 ^2Knight, W. (May 17, 2024). OpenAI's Long-Term AI Risk Team Has Disbanded. *WIRED*.
85. ^Kahn, J. (May 21, 2024). OpenAI promised 20% of its computing power to combat the most dangerous kind of AI, but never delivered, sources say. *Fortune*.
86. ^Amodei, D. (January 2026). The Adolescence of Technology: Confronting and Overcoming the Risks of Powerful AI.
87. ^Rev (January 25, 2026). Musk Speaks at WEF. Transcript of Elon Musk's conversation with Larry Fink at the 2026 World Economic Forum in Davos.
88. ^Corbyn, Z. (June 29, 2024). AI scientist Ray Kurzweil: 'We are going to expand intelligence a millionfold by 2045'. *The Guardian*.
89. ^Aschenbrenner, L. (June 2024). Situational Awareness: The Decade Ahead. Introduction and chapter II, From AGI to Superintelligence: the Intelligence Explosion.
90. ^1 ^2AI Futures Project (August 16, 2026). Q2.5 2026 Timelines Update: Uplift and Revenue. Medians as given in the post's chart descriptions.
91. ^Metaculus. After a (weak) AGI is created, how many months will it be before the first superintelligent AI is created? Question 9062. Community estimate as shown in the Internet Archive capture of July 24, 2026.
92. ^1 ^2Grace, K., et al. Thousands of AI Authors on the Future of AI. 2023 survey; arXiv:2401.02843, version 3.
93. ^1 ^2 ^3 ^4Grace, K., Simonelli, J., Weinstein-Raun, B., Krueger, D., Young, N., Duffy, T., Adamczewski, T., Reeve, J., and Beck, A. (September 2026). Advanced AI according to 1,580 researchers: uncertain, unsafe, and sooner than we thought. AI Impacts. Survey conducted in December 2024.
94. ^AI Futures Project (December 2025). AI Futures Model: Dec 2025 Update.
95. ^AI Futures Project (April 2, 2026). Q1 2026 Timelines Update.
96. ^Bostrom, N. (2012). The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents. *Minds and Machines*, 22(2), pp. 71-85.
97. ^Turner, A. M., et al. Optimal Policies Tend to Seek Power. NeurIPS 2021; version 10 includes subsequent clarifications.
98. ^Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J., and Garrabrant, S. Risks from Learned Optimization in Advanced Machine Learning Systems. Originally released in 2019; arXiv:1906.01820, version 3, December 1, 2021.
99. ^International AI Safety Report 2026. February 2026; section 2.2.2, "Loss of control."
100. ^1 ^2 ^3Independent International Scientific Panel on AI (September 21, 2026). AI Agents, Misalignment and the Risk of Losing Human Control: Evidence from the OpenAI-Hugging Face Incident. Thematic brief, advance unedited version 1. United Nations.
101. ^Hugging Face (July 27, 2026). Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident. Hugging Face blog.
102. ^Greenblatt, R., Cotra, A., and Wijk, H. (August 26, 2026). Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident. METR.
103. ^Salvaggio, E. (September 11, 2026). Rogue AI didn't breach Hugging Face, human decisions did. *Bulletin of the Atomic Scientists* (archived copy).
104. ^1 ^2 ^3 ^4Bengio, Y., et al. (2024). Managing extreme AI risks amid rapid progress. *Science*, 384(6698), pp. 842-845.
105. ^Center for AI Safety. Statement on AI Extinction Risk. Official statement and signatory list; released on May 30, 2023 under the title "Statement on AI Risk" (pre-release archived copy).
106. ^Yudkowsky, E., and Soares, N. (2025). If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. Little, Brown and Company. Publisher's description; publication date September 16, 2025.
107. ^1 ^2Burns, C., et al. (2023). Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. arXiv:2312.09390.
108. ^1 ^2Greenblatt, R., Shlegeris, B., Sachan, K., and Roger, F. (2024). AI Control: Improving Safety Despite Intentional Subversion. ICML 2024.
109. ^1 ^2Hadfield-Menell, D., Dragan, A., Abbeel, P., and Russell, S. (2017). The Off-Switch Game. IJCAI 2017.
110. ^Flynn, F., King, H., and Dragan, A. (September 22, 2025). Strengthening our Frontier Safety Framework. Google DeepMind.
111. ^Altman, S., Brockman, G., and Sutskever, I. (May 22, 2023). Governance of superintelligence. OpenAI (archived copy).
112. ^Hendrycks, D., Schmidt, E., and Wang, A. (March 2025). Superintelligence Strategy. Expert version: arXiv:2503.05628, first posted March 7, 2025.
113. ^Future of Life Institute. Statement on Superintelligence. Official campaign statement, launched October 22, 2025; signature count read October 9, 2026.
114. ^O'Brien, M. (October 22, 2025). Prince Harry, Meghan Markle join with Steve Bannon and Steve Wozniak in calling for ban on AI 'superintelligence' before it destroys the world. The Associated Press, via *Fortune*.
115. ^Future of Life Institute (October 19, 2025). The U.S. Public Wants Regulation (or Prohibition) of Expert-Level and Superhuman AI. Survey of 2,000 US adults conducted by Bryson Gillette, September 29 to October 5, 2025.
116. ^Global Call for AI Red Lines. Campaign website, read October 9, 2026.
117. ^Amodei, D. (September 2026). We Must Pace the Frontier.
118. ^The Associated Press (September 30, 2026). Trump says top tech firms have signed accord to 'self-police' AI development. Via NPR.
119. ^New York Post (September 29, 2026). Trump's 'morally binding' artificial intelligence pledge signed by tech leaders: read in full.
120. ^Bostrom, N. (2026). Optimal Timing for Superintelligence: Mundane Considerations for Existing People. Working paper, version 1.
121. ^Narayanan, A., and Kapoor, S. (April 15, 2025). AI as Normal Technology. Knight First Amendment Institute at Columbia University.
122. ^Kapoor, S., and Narayanan, A. (September 14, 2026). The AI-as-Normal-Technology view of loss-of-control incidents. AI as Normal Technology newsletter.
123. ^Narayanan, A., and Kapoor, S. (October 1, 2026). A big-tent or small-tent AI safety movement? AI as Normal Technology newsletter.
124. ^Heim, A. (March 10, 2026). Yann LeCun's AMI Labs raises $1.03B to build world models. TechCrunch, via Yahoo Finance.
125. ^Lee, L. (March 2025). Report on Yann LeCun's remarks at NVIDIA's GTC conference. *Business Insider*.
126. ^LeCun, Y. (May 18, 2024). Post on X quoting Jan Leike's announcement of his departure from OpenAI. Read via api.fxtwitter.com.
127. ^Marcus, G. (July 28, 2026). Sorry, Sam and Elon, we have not reached the Singularity. Marcus on AI.
128. ^Marcus, G. (September 18, 2026). Wake up, people. What we should actually fear, near term, is not so much rogue superintelligence as unleashed agentic AI causing hacking the internet at scale. Marcus on AI.
129. ^Chollet, F. (November 27, 2017). The implausibility of intelligence explosion. Medium (archived copy).
130. ^Kelly, K. (April 25, 2017). The Myth of a Superhuman AI. *WIRED*.
131. ^AAAI (March 2025). AAAI 2025 Presidential Panel on the Future of AI Research. Association for the Advancement of Artificial Intelligence.
132. ^Gebru, T., and Torres, É. P. (2024). The TESCREAL bundle: Eugenics and the promise of utopia through artificial general intelligence. *First Monday*, 29(4). doi:10.5210/fm.v29i4.13636.
133. ^Gebru, T., and Bender, E. M. (October 1, 2026). Forget 'superintelligence': error-prone AI nearly sparked world war three this month. *The Guardian*.
134. ^Bort, J. (January 26, 2026). AI chip startup Ricursive hits $4B valuation 2 months after launch. *TechCrunch*.
135. ^Meta Superintelligence Labs (July 9, 2026). Introducing Muse Spark 1.1. Meta AI Research.
136. ^Meta Superintelligence Labs (August 10, 2026). Introducing Muse Glimmer: An Open Agentic Model That Runs on Your Device. Meta AI Research.
137. ^Anthropic (September 22, 2026). Claude Opus 5.5 System Card. Section 2, determination on the automated R&D threshold.
138. ^OpenAI (September 29, 2026). Addendum to GPT-6 Astra System Card: GPT-6.1 Sol. OpenAI Deployment Safety Hub.
139. ^Google DeepMind (September 2, 2026). Gemini 3.8 Flash model card. Frontier Safety section.