Future of Humanity Institute (FHI)
Last reviewed
Sources
22 citations
Review status
Source-backed
Revision
v3 · 4,178 words
Improve this article
Add missing citations, update stale details, or suggest a clearer explanation.
Suggest edit
CtrlK
Last reviewed
Sources
22 citations
Review status
Source-backed
Revision
v3 · 4,178 words
Add missing citations, update stale details, or suggest a clearer explanation.
The Future of Humanity Institute (FHI) was a multidisciplinary research institute at the University of Oxford that studied existential risk, the long-term future of humanity, and the transformative potential of advanced technology, with artificial intelligence at the centre of its later work. Founded in November 2005 by the philosopher Nick Bostrom, FHI operated for nineteen years before Oxford's Faculty of Philosophy closed it on 16 April 2024.[1][2] Over its lifetime FHI helped establish existential risk and AI alignment as legitimate academic fields, produced widely cited books including Bostrom's 2014 Superintelligence: Paths, Dangers, Strategies and Toby Ord's 2020 The Precipice, and seeded successor organisations including the Centre for the Governance of AI (GovAI).[3][6]
FHI sat at the intersection of philosophy, computer science, mathematics, economics, and the natural sciences. It was housed within Oxford's Faculty of Philosophy and the Oxford Martin School. Its researchers worked on topics that were unusual for a traditional university department, including the alignment of advanced AI systems, the ethics of acting under deep uncertainty about the far future, anthropic reasoning, the simulation argument, whole-brain emulation, biosecurity, and the governance of frontier technologies. The institute's intellectual output is closely associated with the modern fields of AI safety and effective altruism, and with the broader "longtermist" research programme that has shaped policy debates at the AI Safety Institutes in the United Kingdom and the United States, the EU AI Office, and successive AI Safety Summits.
FHI was established in November 2005 with seed funding drawn from a benefaction of more than 70 million pounds that the IT entrepreneur and futurist James Martin gave to the University of Oxford.[6][20] Martin's gift, then the largest single donation in the university's nine-century history, created the James Martin 21st Century School, later renamed the Oxford Martin School. The Future of Humanity Institute was one of the first research units within that school. By Anders Sandberg's account, the seed grant "was enough to fund a few people for three years" and the institute initially hired three researchers; it survived for nineteen.[6]
Bostrom, a Swedish-born philosopher who had completed his doctorate at the London School of Economics in 2000, had already published Anthropic Bias: Observation Selection Effects in Science and Philosophy in 2002 and his agenda-setting paper "Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards" in the Journal of Evolution and Technology the same year.[7] He used the new institute to develop a research programme around what he called macrostrategy: the study of which actions today would matter most for the long-run trajectory of intelligent life. Anders Sandberg, a Swedish neuroscientist and transhumanist who had collaborated with Bostrom on earlier transhumanist work, joined as one of the institute's first researchers and remained until the closure.
FHI initially occupied modest offices in Littlegate House in central Oxford. Sandberg's Final Report describes the early years as deliberately stripped down, with whiteboards, long open-ended discussions, and a culture that valued unconventional ideas over traditional academic credentialism.[1] Toby Ord, then a graduate student in philosophy who would later co-found Giving What We Can and write The Precipice, was an early affiliate. Other early-period figures included the bioethicist Julian Savulescu, the nanotechnology pioneer Eric Drexler, and the economist Robin Hanson, who spent time at FHI as a visitor.
Nick Bostrom served as the institute's founding director throughout its existence. There was no formal succession plan; the institute's identity was tightly bound to his intellectual agenda and to the small circle of senior researchers around him. Sandberg's Final Report counts roughly eighty staff and visitors over the institute's lifetime, including research fellows, doctoral students, postdoctoral researchers, and short-term visiting scholars.[1]
The table below lists a sample of researchers strongly associated with FHI. Many were also affiliated with adjacent organisations such as the Centre for Effective Altruism, the Global Priorities Institute (founded at Oxford in 2018), and the Machine Intelligence Research Institute.
| Researcher | Role at FHI | Notable contribution |
|---|---|---|
| Nick Bostrom | Founding director, 2005 to 2024 | Superintelligence (2014), simulation argument, vulnerable world hypothesis |
| Anders Sandberg | Senior research fellow, 2006 to 2024 | Whole Brain Emulation: A Roadmap (2008), Grand Futures, FHI Final Report |
| Toby Ord | Senior research fellow | The Precipice (2020), co-founded Giving What We Can |
| Stuart Armstrong | Research fellow | Early AI safety research, oracle AI, value loading |
| Eric Drexler | Research fellow (period affiliation) | Atomically precise manufacturing, nanotechnology risk |
| Carl Shulman | Research associate | AI macroeconomics, takeoff dynamics |
| Allan Dafoe | Director of governance research | Founded the Centre for the Governance of AI within FHI |
| Owain Evans | Research fellow | AI truthfulness, language model evaluation, TruthfulQA |
| Vincent Müller | Research fellow | Philosophy of AI, AI ethics surveys |
| Robin Hanson | Visiting researcher | The Age of Em, prediction markets |
| Cecilia Tilli | Project manager and researcher | Existential risk communication |
| Julian Savulescu | Affiliated bioethicist | Human enhancement ethics |
| William MacAskill | Informal collaborator | What We Owe the Future, longtermism |
Demis Hassabis, the co-founder and CEO of DeepMind, sat on FHI's scientific advisory board in the institute's middle years, illustrating the early traffic between FHI and the AI laboratories that later became central to the field.[3]
FHI's research portfolio was deliberately broad and shifted over time. In the institute's first decade, anthropic reasoning, human enhancement, and global catastrophic risks were prominent. From around 2010 onward, AI safety and the governance of advanced AI systems became a primary focus. The table below summarises the main research strands.
| Research area | Description |
|---|---|
| Existential risk | The general framework set out in Bostrom's 2002 paper, defining risks that threaten the premature extinction of Earth-originating intelligent life or the permanent and drastic destruction of its potential for desirable future development |
| AI safety and alignment | Technical and philosophical work on how to build advanced AI systems whose behaviour aligns with human values, including work on oracle AI, value learning, mesa-optimisation, and language model evaluations |
| Existential risk from AI | Sub-area of existential risk focused specifically on misaligned or misused advanced AI; see existential risk from AI |
| Global catastrophic biological risks | Research on engineered pandemics, dual-use biotechnology, and biosecurity governance |
| Whole-brain emulation | Sandberg and Bostrom's 2008 roadmap for scanning and computationally simulating a complete human brain |
| Anthropic reasoning | Observation-selection effects, the doomsday argument, and the simulation argument |
| Nanotechnology risk | Drexler-influenced work on the long-run risks and governance of atomically precise manufacturing |
| AI governance | Policy, strategy, and institutional design for advanced AI; eventually spun off as the Centre for the Governance of AI under Allan Dafoe |
| Macrostrategy and cause prioritisation | Methodology for identifying which interventions most affect the long-run future, closely linked to the effective altruism research community |
| Astronomical-scale considerations | Work on cosmic endowments, astronomical waste, grand futures, and the long reflection |
| Information hazards | Bostrom's framework for risks that arise from the dissemination of true information |
FHI produced a steady stream of academic papers, technical reports, and books. Several of these became foundational texts for the modern fields of AI safety, existential risk studies, and longtermism.
| Year | Author or authors | Title and venue | Contribution |
|---|---|---|---|
| 2002 | Nick Bostrom | "Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards", Journal of Evolution and Technology | Introduced the modern concept of existential risk and a typology of catastrophe scenarios |
| 2002 | Nick Bostrom | Anthropic Bias: Observation Selection Effects in Science and Philosophy (Routledge) | Foundational treatment of anthropic reasoning |
| 2003 | Nick Bostrom | "Are You Living in a Computer Simulation?", Philosophical Quarterly | The simulation argument, widely cited in popular and academic discussions |
| 2008 | Anders Sandberg and Nick Bostrom | Whole Brain Emulation: A Roadmap, FHI Technical Report 2008-3 | First detailed technical roadmap for mind uploading |
| 2008 | Nick Bostrom and Milan Cirkovic (eds.) | Global Catastrophic Risks (Oxford University Press) | Edited volume that helped consolidate the field |
| 2009 | Julian Savulescu and Nick Bostrom (eds.) | Human Enhancement (Oxford University Press) | Influential volume on the ethics of enhancement |
| 2011 | Nick Bostrom | "Infinite Ethics", Analysis and Metaphysics | Decision theory under infinite value |
| 2011 | Nick Bostrom | "Information Hazards: A Typology of Potential Harms from Knowledge", Review of Contemporary Philosophy | Coined the term "information hazards" |
| 2014 | Nick Bostrom | Superintelligence: Paths, Dangers, Strategies (Oxford University Press) | New York Times bestseller credited with bringing AI existential risk into mainstream discussion |
| 2018 | Allan Dafoe | AI Governance: A Research Agenda, FHI report | Set the agenda for the AI governance subfield |
| 2019 | Nick Bostrom | "The Vulnerable World Hypothesis", Global Policy | Argues that some future technologies could be "black balls" that destroy civilisation by default |
| 2020 | Toby Ord | The Precipice: Existential Risk and the Future of Humanity (Bloomsbury) | Estimates a one in six chance of existential catastrophe this century, with one in ten attributed to misaligned AI |
| 2024 | Anders Sandberg | Future of Humanity Institute 2005-2024: Final Report, Oxford University Research Archive | Oral history of the institute, written after its closure |
A number of FHI papers also entered policy and academic discourse without becoming books. These include work on the unilateralist's curse (Bostrom, Douglas, and Sandberg, 2016), Bostrom's writing on moral uncertainty, and a long series of technical AI safety papers from researchers including Stuart Armstrong, Owain Evans, and visiting collaborators.
Bostrom's Superintelligence deserves separate mention. Published in 2014, it was endorsed on its dust jacket by Bill Gates, Stuart Russell, and Martin Rees, and reached the New York Times bestseller list, where it ranked among the best-selling science titles for August 2014.[10][21] Elon Musk publicly cited the book when he warned that AI was "potentially more dangerous than nuclear weapons", and in 2015 Musk provided 1 million pounds to FHI for AI risk research through the Future of Life Institute.[5][20] Stephen Hawking joined Bostrom and others as a signatory on the 2015 Future of Life Institute open letter on research priorities for robust and beneficial AI. The book is widely regarded as the moment that AI existential risk moved from a niche concern of a few philosophers and rationalists to a topic discussed by mainstream scientists, technologists, and policy makers.
In The Precipice (2020), Toby Ord, a senior research fellow at FHI, put humanity's total risk of existential catastrophe over the next century at roughly one in six.[12][22] Ord assigned the single largest share of that risk, about one in ten, to unaligned artificial general intelligence, which he judged greater than all other sources of existential risk combined; by comparison he estimated the existential risk from nuclear war this century at about one in a thousand.[12][22] Ord stressed that the figures were intended as order-of-magnitude judgements rather than precise probabilities. The book became a defining text for the existential risk literature and is frequently cited alongside Superintelligence as evidence of FHI's outsized influence on how the field frames catastrophic and existential risk from AI.
FHI's influence on the AI field is hard to overstate, even though the institute itself was small and produced no AI systems. Several patterns recur in the historical record.
First, FHI helped legitimise AI existential risk as a research topic. Before the early 2010s, the idea that advanced AI might pose a catastrophic or existential threat was largely confined to a handful of writers including Eliezer Yudkowsky and a small online community around the Singularity Institute (later the Machine Intelligence Research Institute, MIRI). Bostrom's academic credentials, the institutional cover provided by Oxford, and the technical seriousness of Superintelligence changed the social standing of the topic.
Second, FHI alumni and affiliates moved into senior positions across the AI safety ecosystem. Former FHI researchers and visitors went on to work at OpenAI, DeepMind, Anthropic, the Machine Intelligence Research Institute, the Center for AI Safety, Conjecture, and the AI safety teams of national governments. Allan Dafoe became head of long-term strategy and governance at Google DeepMind while continuing as president of GovAI.[15] Owain Evans's group worked on language model evaluation. Toby Ord advised the World Health Organization, the United Nations Secretary-General's office, and the UK government on existential risk policy.
Third, FHI's framing was visible in the founding statements of major AI organisations. Sam Altman and Elon Musk both cited Bostrom in early discussions of OpenAI's mission. Demis Hassabis served on FHI's advisory board. The 2015 Puerto Rico AI safety conference, the 2017 Asilomar Principles, and later the 2023 Bletchley Declaration at the UK AI Safety Summit all drew on a vocabulary of catastrophic and existential risk that FHI had helped to develop.
Fourth, FHI seeded successor organisations. The Centre for the Governance of AI (GovAI), founded by Dafoe within FHI in 2018, became an independent non-profit in 2021 after the opportunity arose for Dafoe to take on a senior role at DeepMind; the university would not allow him to hold a dual appointment.[15][16] The Global Priorities Institute, founded at Oxford in 2018 by William MacAskill and Hilary Greaves, overlapped substantially with FHI's macrostrategy work and continued after FHI's closure. The Forethought Foundation for Global Priorities Research and the Macrostrategy Research Initiative also drew on FHI personnel.
FHI was funded through a mix of sources that shifted across its lifetime. Initial support came from James Martin's gift to what became the Oxford Martin School.[6][20] Later donors included the European Research Council, the Leverhulme Trust, the insurance company Amlin, the Future of Life Institute (a separate organisation co-founded by Max Tegmark and Jaan Tallinn that shares ideological roots but is institutionally distinct), Open Philanthropy (which recommended a 13.4 million pound grant in 2018), and Elon Musk's 1 million pound 2015 contribution to AI risk research.[5] By the late 2010s a substantial share of the institute's funding came from donors associated with the effective altruism movement.
Funding was not the proximate cause of the closure. Sandberg's Final Report and contemporaneous reporting indicate that, at the time of the closure, FHI had committed funds and outside donors willing to provide more.[1][4] The constraint was administrative rather than financial.
In early January 2023, Bostrom posted on his personal website a document titled "Apology for an Old Email" in which he addressed an email he had sent in 1996 to the Extropians mailing list as a graduate student.[13] The 1996 email contained the sentence "Blacks are more stupid than whites" and used a racial slur in describing how he expected the statement to be received. Bostrom said he had been alerted that an archive of the list was being searched and wanted to get ahead of the discovery. He described the original email as "disgusting" and said he "completely" repudiated it, though parts of his apology, including comments on intelligence and genetics, were widely criticised as inadequate.[13]
The University of Oxford opened an investigation. The investigation concluded on 10 August 2023 with a statement that the university did "not consider you to be a racist or that you hold racist views" and that it considered Bostrom's January 2023 apology to have been sincere.[14] The episode generated extensive coverage in The Daily Beast, The Oxford Student, The Oxford Blue, and elsewhere, and prompted debate within the effective altruism community about how to respond.
Reporting on the FHI closure repeatedly notes that the controversy contributed to administrative friction between the institute and the Faculty of Philosophy, although neither the university nor Bostrom has issued a definitive statement linking the two.[4][5]
On 16 April 2024 the Faculty of Philosophy at Oxford announced that the Future of Humanity Institute had been closed.[1][4] The university's official statement said that after consideration of "the best structures for conducting our academic research" the decision had been made to close FHI, while recognising "the Institute's important contribution to this emerging field, which researchers elsewhere across the University are likely to continue".[5]
The sequence of events leading to the closure, as described by Sandberg in his Final Report and in subsequent reporting, was as follows.[1][6] Beginning in 2020, the Faculty of Philosophy imposed a freeze on hiring and on new fundraising at FHI. The reasons for that freeze were not made public. Senior researchers and junior staff began leaving as their contracts ended and could not be renewed. The Centre for the Governance of AI separated and became an independent non-profit in 2021. In late 2023, the Faculty informed the institute that the contracts of remaining FHI staff would not be renewed at all. Bostrom resigned from the University of Oxford in connection with the closure.[5]
Bostrom characterised the situation as "a death by bureaucracy", explaining that "we were administratively housed within the faculty of philosophy, even though the majority of our research team by this time were non-philosophers".[5] Sandberg used the phrase "gradual suffocation by Faculty bureaucracy".[1] To illustrate the administrative load, FHI staff coined a joke measurement unit called "the Oxford", described in the Final Report and contemporaneous coverage "as a shorthand for the amount of work it takes to read and write 308 emails", which was the actual effort required to have a single small grant disbursed into the institute's account after both the funder and the university had already approved it.[6] His Final Report, hosted on the Oxford University Research Archive, doubles as an oral history of the institute, an account of its scholarly contributions, and a set of lessons on running unconventional research groups inside traditional universities.[1]
FHI's intellectual programme did not end with the institute. Several organisations now carry forward overlapping research agendas.
| Organisation | Founded | Relationship to FHI |
|---|---|---|
| Centre for the Governance of AI (GovAI) | 2018 within FHI; independent non-profit from 2021 | Direct spin-off; led by Allan Dafoe, then Ben Garfinkel |
| Global Priorities Institute (GPI), Oxford | 2018 | Overlapping macrostrategy and longtermism research; founded by William MacAskill and Hilary Greaves |
| Forethought Foundation for Global Priorities Research | 2018 (closed 2024) | Funded longtermist academic work, including by FHI affiliates |
| Macrostrategy Research Initiative | 2024 | Founded by former FHI researchers after closure |
| Center for AI Safety | 2022 | Independent US non-profit; many staff trained or influenced by FHI work |
| Machine Intelligence Research Institute (MIRI) | 2000 (originally Singularity Institute) | Independent but ideologically and personally intertwined with FHI |
| Anthropic, OpenAI, Google DeepMind safety teams | various | Employ many former FHI staff and visitors |
| UK AI Safety Institute | 2023 | Established after the Bletchley AI Safety Summit; intellectual lineage from FHI's existential risk framing |
The Future of Life Institute, often confused with FHI, is a separate organisation founded in the United States in 2014 by Max Tegmark, Jaan Tallinn, Anthony Aguirre, Viktoriya Krakovna, and Meia Chita-Tegmark. FHI and FLI shared collaborators and intellectual concerns but were institutionally distinct.
FHI was not without critics, and the criticism intensified as longtermism became more prominent in public debate. The main lines of critique are summarised below.
The most sustained external critic has been the philosopher Émile P. Torres, formerly Phil Torres, who once wrote in defence of longtermism but later argued that it is "quite possibly the most dangerous secular belief system in the world today".[17] Writing in Salon in August 2022 and in subsequent academic and journalistic work, Torres argued that longtermism is rooted in transhumanist and eugenicist intellectual traditions and that its emphasis on enormous future populations risks justifying the sacrifice of present people for far-future projects.[17] With the AI ethics researcher Timnit Gebru, Torres proposed the acronym TESCREAL (transhumanism, extropianism, singularitarianism, cosmism, rationalism, effective altruism, longtermism) to describe what they see as a related cluster of ideologies, several of which are tightly linked to FHI's intellectual circle.
Other critics have argued that the longtermist framework underweights near-term harms in favour of speculative far-future considerations, that it treats consequentialist calculations over very long horizons as more reliable than they actually are, and that the social composition of the field skews toward wealthy donors and a narrow demographic. Writers including Erik Hoel and the AI ethics researcher Kate Crawford have raised related concerns, although their critiques target the broader effective altruism and longtermism movements rather than FHI specifically. Some defenders within the field, including MacAskill, responded by reformulating longtermism in weaker forms that do not require the strongest assumptions about aggregating value across vast populations.
The 2023 Bostrom email controversy drew its own substantial body of criticism, both from outside commentators and from within the effective altruism community. Reactions on the Effective Altruism Forum and elsewhere were divided between those who accepted Bostrom's apology, those who found it inadequate, and those who argued that the entire episode reflected longstanding problems with the social culture around FHI and adjacent groups.
FHI's institutional life was nineteen years. Its intellectual influence is likely to last considerably longer.
The modern field of AI safety, both in its technical and governance forms, is in large part the product of work that originated at or passed through FHI. Researchers trained or affiliated with the institute now hold senior positions at AI laboratories including Anthropic, OpenAI, and Google DeepMind, and at government bodies including the UK AI Safety Institute, the US AI Safety Institute, and the EU AI Office. The vocabulary of existential risk, AI alignment, information hazards, the unilateralist's curse, the vulnerable world hypothesis, and longtermism is in everyday use in policy and research discussions that did not exist in their current form before FHI began publishing.
Superintelligence remains foundational reading in the field a decade after publication, and Ord's The Precipice has played a similar role for the broader existential risk literature.[10][12] Sandberg's Final Report, posted on the Oxford University Research Archive in April 2024, has been read both as an obituary for the institute and as a manual for future research groups attempting work that does not fit neatly inside traditional university structures.[1]
The closure of FHI also raised harder questions about whether the modern university is well suited to host long-horizon, interdisciplinary, agenda-setting research of the kind FHI tried to do. Several proposals for successor institutes outside Oxford, by former staff and outside observers, are now in early stages.