Last reviewed
Sources
No citations yet
Review status
Needs citations
Revision
v3 ยท 3,644 words
Add missing citations, update stale details, or suggest a clearer explanation.
The Centre for the Governance of AI (GovAI) is an independent nonprofit research organisation that studies how governments, AI developers, and other actors should manage the transition to a world with advanced artificial intelligence. It was founded in 2018 by political scientist Allan Dafoe as a research programme inside the University of Oxford's future of humanity institute, and operated as a unit of that institute until spinning out as a separate nonprofit in 2021.[1][2] Following the closure of FHI by Oxford in April 2024, GovAI continued as a stand-alone organisation based in Oxford with a Bay Area presence and remote staff.[3][4] The centre is led by Executive Director Ben Garfinkel, with Allan Dafoe serving as president and Markus Anderljung as Director of Policy and Research.[5][6][7] GovAI has produced influential reports on frontier-model regulation, compute governance, and international AI agreements, and its researchers contributed to the framing of the 2023 UK AI Safety Summit at Bletchley Park.[8][9][10]
The seed of GovAI was a small AI governance research group inside the University of Oxford's Future of Humanity Institute, an interdisciplinary research centre in the Faculty of Philosophy directed by philosopher nick bostrom.[11] Political scientist Allan Dafoe, then an associate professor of the International Politics of AI at Oxford, joined FHI in the late 2010s and formalised the AI governance work as a programme during 2017 to 2018.[1][2] FHI's 2018 annual report identifies the unit as the "Center for the Governance of AI" and lists Dafoe as its director.[1] The same year Dafoe published a 75-page document titled "AI Governance: A Research Agenda", a foundational text that mapped the field into questions about strategic competition, international cooperation, technical landscape, and governance institutions for advanced AI.[2]
The unit's early staff and affiliates produced one of the most cited reports in the field, "The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation", led by Miles Brundage of FHI and Shahar Avin of the Centre for the Study of Existential Risk, with contributors from OpenAI, the Electronic Frontier Foundation, and other institutions.[12] The report was released in February 2018 and helped frame public debate about misuse risks from machine learning systems.[12]
Between 2018 and 2021 the Centre for the Governance of AI expanded as the AI governance research programme inside FHI. It built up a network of research affiliates across political science, law, economics, and computer science, and ran the Governance of AI Fellowship, an early summer research programme that recruited junior researchers into the field.[13] Output during this period included Dafoe's 2018 research agenda; commentary on the international AI landscape; work on AI capability forecasting; and contributions to safety-related publications by collaborators inside OpenAI, DeepMind, and other labs.[2][14]
By 2020 the centre had grown to around a dozen researchers and affiliates. Within FHI itself, however, administrative tensions with the Faculty of Philosophy were intensifying. From 2020 the Faculty imposed a freeze on FHI's fundraising and hiring, a constraint that the institute's eventual final report described as having gradually suffocated its operations.[4][15]
In September 2021 GovAI announced that it would spin out of FHI and operate as an independent organisation, citing a desire for greater operational flexibility, the ability to expand its field-building activities, and the bureaucratic constraints affecting its parent institute.[16] The transition was completed in October 2021. A blog post on the GovAI site dated 29 October 2021 announced the relaunch, naming Ben Garfinkel as acting director and Allan Dafoe as president.[5] At that point Dafoe had also taken up a position leading a long-term AI strategy and governance team at DeepMind, with the GovAI presidency held in a part-time capacity.[5]
The new organisation was structured around two legal entities: a UK company limited by guarantee, the Centre for the Governance of AI Ltd., and a US 501(c)(3) nonprofit, Centre for the Governance of AI, Inc.[17] Both entities operate under the GovAI brand, with most staff based in Oxford and a smaller cohort in the San Francisco Bay Area and other locations.[17][18]
On 16 April 2024 the University of Oxford formally closed the Future of Humanity Institute. The official statement cited "increasing administrative headwinds within the Faculty of Philosophy" as a primary cause; the closure was preceded by a 2023 decision by the Faculty not to renew the contracts of remaining FHI staff.[3][4] FHI's final report, published on the EA Forum, framed the wind-down as the result of a long-running clash between the institute's flexible operating style and the rules of its host faculty.[4]
Because GovAI had already become a separate legal entity in 2021, the FHI closure did not directly terminate its operations. Coverage of the FHI closure in Asterisk magazine and the LessWrong forum identified GovAI as one of the most significant continuing offshoots of FHI's AI governance work, alongside the broader Oxford research community.[15][19] GovAI continued to operate from Oxford and announced expanded hiring and fellowship rounds in 2024 and 2025, including a 2026 Summer Fellowship with research and applied tracks.[18][20]
Allan Dafoe is the founder and president of GovAI. He completed a PhD in political science at the University of California, Berkeley, and held faculty positions at Yale University before moving to Oxford. At Oxford he was associate professor of the International Politics of AI, with a position in the Department of Politics and International Relations and at FHI, where he founded and directed the AI governance unit that became GovAI.[1][2] Dafoe's earlier academic work focused on causes of war, the democratic peace, and methodology in international relations, with awards including the Kenneth E. Boulding Award.[21] Since 2020 he has led a long-term AI strategy and governance team at Google DeepMind in addition to his GovAI role.[21][5]
Ben Garfinkel is the executive director of GovAI, named acting director at the 2021 relaunch and confirmed as executive director thereafter.[5][6] He holds undergraduate degrees in Physics and in Mathematics and Philosophy from Yale University, and a DPhil in International Relations from Oxford. His thesis examined the role of economic growth in international security.[6] Garfinkel's research has focused on the security implications of AI, the causes of war, and forecasting methodology for emerging technologies; he has been a research fellow at FHI alongside his GovAI work, and has commented publicly on the design of global AI safety summits and AGI safety research agendas.[6]
Markus Anderljung is Director of Policy and Research at GovAI. His research focuses on frontier AI regulation, compute governance, third-party auditing of AI systems, and international AI policy.[7][22] Anderljung was seconded to the UK Cabinet Office as a Senior AI Policy Specialist during the lead-up to the 2023 AI Safety Summit at Bletchley Park, and has served as an adjunct fellow at the Center for a New American Security and as a member of the OECD AI Policy Observatory's Expert Group on AI Futures.[7][22] He was selected as a vice-chair of the drafting process for the European Union's Code of Practice for general-purpose AI under the EU AI Act.[7]
GovAI's senior staff include international governance lead Robert Trager, who is also co-director of the Oxford Martin AI Governance Initiative and a senior research fellow at the Blavatnik School of Government.[23] Toby Shevlane, a researcher with positions at GovAI and Google DeepMind, has led work on model evaluations for extreme risks and on structured access to powerful AI models.[24][25] Helen Toner, formerly of Georgetown's Center for Security and Emerging Technology, has been listed among GovAI's senior staff and advisory roles, and is a member of the advisory board.[26][16]
The advisory board, formalised at the 2021 relaunch, has included Ajeya Cotra of Open Philanthropy, Allan Dafoe in a board capacity, Helen Toner, Tasha McCauley, and Toby Ord, with composition updated periodically. Toby Ord, a senior research fellow at Oxford and a co-founder of Giving What We Can, is one of the longest-serving advisers.[16][27]
GovAI's research programme covers a set of overlapping AI governance topics, generally organised into policy and regulation, technical governance, AI economics, and the broader AI landscape.[28]
Frontier model regulation has been a flagship area since 2022. Anderljung, Garfinkel, Shevlane, Trager, and others have argued that highly capable foundation models pose distinctive policy challenges because dangerous capabilities can emerge unexpectedly, misuse is difficult to prevent after deployment, and models can proliferate widely once released.[8][29] GovAI has proposed regulatory components including standard-setting processes, registration and reporting requirements, and compliance mechanisms for frontier model forum participants and other frontier developers.[8] The work has fed into uk aisi and us aisi policy discussions and the design of the eu ai act general-purpose AI provisions.[7][9]
Work on compute governance has been led by Anderljung, Lennart Heim, Robert Trager, and collaborators across multiple institutions. The February 2024 paper "Computing Power and the Governance of AI" co-authored by 19 researchers including Yoshua Bengio, Diane Coyle, and Gillian Hadfield argued that compute is detectable, excludable, and quantifiable across a concentrated supply chain, making it a tractable lever for governance.[29] The paper outlined uses of compute governance for visibility into AI development, allocation toward beneficial uses, and restriction of harmful uses, while warning of risks to privacy and power concentration.[29]
Shevlane led the 2023 paper "Model evaluation for extreme risks", co-authored with 20 collaborators including Yoshua Bengio, Paul Christiano, Allan Dafoe, and researchers at Google DeepMind, OpenAI, Anthropic, and academic institutions.[25] The paper proposed that frontier developers run two complementary types of evaluation: dangerous capability evaluations to detect harmful functionalities such as offensive cyber, autonomous replication, or persuasion, and alignment evaluations to assess whether models would apply those capabilities for harm.[25] The framework has been cited in evaluation protocols at apollo research, redwood research, and metr, and in policy commentary by AI safety institutes.[25]
The international governance workstream is led by Robert Trager and Allison Carnegie, with Lewis Ho, Lennart Heim, and others as co-authors. A flagship paper, "International governance of civilian AI: a jurisdictional certification approach", drew analogies to the International Civil Aviation Organization, the International Maritime Organization, and the Financial Action Task Force, arguing that a network of national regulators could certify jurisdictions rather than firms directly.[23] Other work covers verification of international AI commitments, treaty design, and risks from unconstrained competition between states and labs.[23]
GovAI has continued the strand of forecasting work begun under FHI, addressing AGI timelines, race dynamics, and broader strategy questions. Garfinkel and Dafoe have written on the dynamics of strategic competition and the conditions under which cooperative governance is feasible, and the organisation has hosted workshops on capability forecasting jointly with collaborators inside major labs.[2][14]
The economics workstream, with contributors including Anton Korinek (an affiliate and frequent co-author) and Diane Coyle, has examined labour-market impacts of large language models, the macroeconomics of transformative AI, and benefit-sharing arrangements.[28] GovAI co-hosts the Brookings/GovAI Conference on the Governance of Transformative AI with Brookings researchers.[30]
GovAI also publishes shorter analysis pieces and policy briefs through its website and through outlets such as Lawfare, Foreign Affairs, and the Financial Times.[17][31]
GovAI runs a Summer Fellowship that has operated annually since the FHI period and has become one of the principal entry routes into the AI governance research community.[13][20] The contemporary version of the programme, branded the GovAI Summer Fellowship, has both a Research Track and an Applied Track. The 2026 edition ran from 8 June to 28 August 2026, was based in London for the in-person cohort, and paid a stipend of GBP 12,000 over the three-month period.[20] Fellows in the Research Track spend the opening weeks exploring the landscape before settling on an independent research proposal with mentorship from GovAI staff; outputs include policy briefs, journal articles, and white papers.[20]
GovAI also runs longer-term roles for early-career researchers, including a Research Scholar programme that can lead to staff Research Fellow positions, and offers one-year research fellowships in defined policy and technical governance areas.[32][33] The organisation supports affiliates based at universities, think tanks, and AI labs as part of its broader field-building activities.[32]
GovAI is funded by philanthropic grants and donations. The 2021 spin-off announcement listed Open Philanthropy, the Center for Emerging Risk Research, and Effective Altruism Funds as principal funders.[16] In subsequent years open philanthropy has been GovAI's largest disclosed funder. Open Philanthropy's grant database records a USD 2,537,600 two-year grant to support field-building activities in AI governance research, made via the Centre for Effective Altruism, and at least two further general-support grants of USD 2,800,000 and USD 1,000,000 to GovAI directly, along with a smaller compute strategy workshop grant.[34][35][36]
GovAI is associated with the effective altruism philanthropic ecosystem through its funders and the prior affiliations of its founders and advisers, though as an independent registered charity it sets its own research priorities.[16][17]
GovAI research has been cited in official government documents, multilateral processes, and major news coverage of AI policy. The UK government's 2023 AI Safety Summit at Bletchley Park drew on GovAI-affiliated researchers, and Anderljung's secondment to the Cabinet Office contributed directly to the summit's policy framing.[9][7] The Bletchley Declaration and the chair's statement from the summit referenced frontier AI risks in ways that overlap closely with the categories proposed in the "Frontier AI Regulation" paper.[9][10]
Within the European Union, GovAI staff and affiliates have provided input to consultations on the AI Act, including the design of the Code of Practice for general-purpose AI, where Anderljung was named a vice-chair of one of the drafting working groups.[7] In the United States, GovAI work has been cited in policy commentary around the 2023 White House ai executive order on AI and subsequent administration AI policies, and in materials produced by the us aisi and the uk aisi.[7][25]
GovAI's writing appears in academic journals, think-tank reports, and outlets including Foreign Affairs, The Economist, the Financial Times, Lawfare, Nature Machine Intelligence, and Science.[17][7]
| Organisation | Founded | Headquarters | Primary focus | Structure |
|---|---|---|---|---|
| Centre for the Governance of AI (GovAI) | 2018 (spun out 2021) | Oxford, UK and US | AI governance research and policy | Independent nonprofit[17] |
| center for ai safety | 2022 | San Francisco, US | AI safety research and field-building | Independent nonprofit[37] |
| apollo research | 2023 | London, UK | Model evaluations and deceptive alignment | Independent nonprofit[38] |
| metr | 2022 | Berkeley, US | Autonomy and capability evaluations | Independent nonprofit[39] |
| uk aisi | 2023 | London, UK | National AI safety institute | UK government body[40] |
| us aisi | 2023 | Gaithersburg, US | National AI safety institute | US government body within NIST[41] |
| frontier model forum | 2023 | US | Industry forum for frontier developers | Industry nonprofit[42] |
GovAI is distinct from the AI Safety Institutes in that it is non-governmental, and from organisations such as apollo research and metr in that it focuses on governance research and policy rather than running model evaluations directly. Its work intersects with center for ai safety in field-building and policy commentary, though CAIS places greater emphasis on technical safety research. Compared to industry coordination bodies such as the frontier model forum, GovAI is independent of AI developers, though its researchers frequently co-author papers with personnel at frontier labs.[17][29]