Future of Humanity Institute (FHI)

The Future of Humanity Institute (FHI) was a multidisciplinary research institute at the University of Oxford that studied existential risk, the long-term future of humanity, and the transformative potential of advanced technology, with artificial intelligence at the centre of its later work. Founded in November 2005 by the philosopher Nick Bostrom, FHI operated for nineteen years before Oxford's Faculty of Philosophy closed it on 16 April 2024.^[1][2] Over its lifetime FHI helped establish existential risk and AI alignment as legitimate academic fields, produced widely cited books including Bostrom's 2014 Superintelligence: Paths, Dangers, Strategies and Toby Ord's 2020 The Precipice, and seeded successor organisations including the Centre for the Governance of AI (GovAI).^[3][6]

FHI sat at the intersection of philosophy, computer science, mathematics, economics, and the natural sciences. It was housed within Oxford's Faculty of Philosophy and the Oxford Martin School. Its researchers worked on topics that were unusual for a traditional university department, including the alignment of advanced AI systems, the ethics of acting under deep uncertainty about the far future, anthropic reasoning, the simulation argument, whole-brain emulation, biosecurity, and the governance of frontier technologies. The institute's intellectual output is closely associated with the modern fields of AI safety and effective altruism, and with the broader "longtermist" research programme that has shaped policy debates at the AI Safety Institutes in the United Kingdom and the United States, the EU AI Office, and successive AI Safety Summits.

When was the Future of Humanity Institute founded?

FHI was established in November 2005 with seed funding drawn from a benefaction of more than 70 million pounds that the IT entrepreneur and futurist James Martin gave to the University of Oxford.^[6][20] Martin's gift, then the largest single donation in the university's nine-century history, created the James Martin 21st Century School, later renamed the Oxford Martin School. The Future of Humanity Institute was one of the first research units within that school. By Anders Sandberg's account, the seed grant "was enough to fund a few people for three years" and the institute initially hired three researchers; it survived for nineteen.^[6]

Bostrom, a Swedish-born philosopher who had completed his doctorate at the London School of Economics in 2000, had already published Anthropic Bias: Observation Selection Effects in Science and Philosophy in 2002 and his agenda-setting paper "Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards" in the Journal of Evolution and Technology the same year.^[7] He used the new institute to develop a research programme around what he called macrostrategy: the study of which actions today would matter most for the long-run trajectory of intelligent life. Anders Sandberg, a Swedish neuroscientist and transhumanist who had collaborated with Bostrom on earlier transhumanist work, joined as one of the institute's first researchers and remained until the closure.

FHI initially occupied modest offices in Littlegate House in central Oxford. Sandberg's Final Report describes the early years as deliberately stripped down, with whiteboards, long open-ended discussions, and a culture that valued unconventional ideas over traditional academic credentialism.^[1] Toby Ord, then a graduate student in philosophy who would later co-found Giving What We Can and write The Precipice, was an early affiliate. Other early-period figures included the bioethicist Julian Savulescu, the nanotechnology pioneer Eric Drexler, and the economist Robin Hanson, who spent time at FHI as a visitor.

Leadership and people

Nick Bostrom served as the institute's founding director throughout its existence. There was no formal succession plan; the institute's identity was tightly bound to his intellectual agenda and to the small circle of senior researchers around him. Sandberg's Final Report counts roughly eighty staff and visitors over the institute's lifetime, including research fellows, doctoral students, postdoctoral researchers, and short-term visiting scholars.^[1]

The table below lists a sample of researchers strongly associated with FHI. Many were also affiliated with adjacent organisations such as the Centre for Effective Altruism, the Global Priorities Institute (founded at Oxford in 2018), and the Machine Intelligence Research Institute.

Scroll sideways for more →

Demis Hassabis, the co-founder and CEO of DeepMind, sat on FHI's scientific advisory board in the institute's middle years, illustrating the early traffic between FHI and the AI laboratories that later became central to the field.^[3]

What did the Future of Humanity Institute research?

FHI's research portfolio was deliberately broad and shifted over time. In the institute's first decade, anthropic reasoning, human enhancement, and global catastrophic risks were prominent. From around 2010 onward, AI safety and the governance of advanced AI systems became a primary focus. The table below summarises the main research strands.

Scroll sideways for more →

Key publications and ideas

FHI produced a steady stream of academic papers, technical reports, and books. Several of these became foundational texts for the modern fields of AI safety, existential risk studies, and longtermism.

Scroll sideways for more →

A number of FHI papers also entered policy and academic discourse without becoming books. These include work on the unilateralist's curse (Bostrom, Douglas, and Sandberg, 2016), Bostrom's writing on moral uncertainty, and a long series of technical AI safety papers from researchers including Stuart Armstrong, Owain Evans, and visiting collaborators.

Bostrom's Superintelligence deserves separate mention. Published in 2014, it was endorsed on its dust jacket by Bill Gates, Stuart Russell, and Martin Rees, and reached the New York Times bestseller list, where it ranked among the best-selling science titles for August 2014.^[10][21] Elon Musk publicly cited the book when he warned that AI was "potentially more dangerous than nuclear weapons", and in 2015 Musk provided 1 million pounds to FHI for AI risk research through the Future of Life Institute.^[5][20] Stephen Hawking joined Bostrom and others as a signatory on the 2015 Future of Life Institute open letter on research priorities for robust and beneficial AI. The book is widely regarded as the moment that AI existential risk moved from a niche concern of a few philosophers and rationalists to a topic discussed by mainstream scientists, technologists, and policy makers.

How did The Precipice estimate existential risk?

In The Precipice (2020), Toby Ord, a senior research fellow at FHI, put humanity's total risk of existential catastrophe over the next century at roughly one in six.^[12][22] Ord assigned the single largest share of that risk, about one in ten, to unaligned artificial general intelligence, which he judged greater than all other sources of existential risk combined; by comparison he estimated the existential risk from nuclear war this century at about one in a thousand.^[12][22] Ord stressed that the figures were intended as order-of-magnitude judgements rather than precise probabilities. The book became a defining text for the existential risk literature and is frequently cited alongside Superintelligence as evidence of FHI's outsized influence on how the field frames catastrophic and existential risk from AI.

How did the Future of Humanity Institute influence the AI community?

FHI's influence on the AI field is hard to overstate, even though the institute itself was small and produced no AI systems. Several patterns recur in the historical record.

First, FHI helped legitimise AI existential risk as a research topic. Before the early 2010s, the idea that advanced AI might pose a catastrophic or existential threat was largely confined to a handful of writers including Eliezer Yudkowsky and a small online community around the Singularity Institute (later the Machine Intelligence Research Institute, MIRI). Bostrom's academic credentials, the institutional cover provided by Oxford, and the technical seriousness of Superintelligence changed the social standing of the topic.

Second, FHI alumni and affiliates moved into senior positions across the AI safety ecosystem. Former FHI researchers and visitors went on to work at OpenAI, DeepMind, Anthropic, the Machine Intelligence Research Institute, the Center for AI Safety, Conjecture, and the AI safety teams of national governments. Allan Dafoe became head of long-term strategy and governance at Google DeepMind while continuing as president of GovAI.^[15] Owain Evans's group worked on language model evaluation. Toby Ord advised the World Health Organization, the United Nations Secretary-General's office, and the UK government on existential risk policy.

Third, FHI's framing was visible in the founding statements of major AI organisations. Sam Altman and Elon Musk both cited Bostrom in early discussions of OpenAI's mission. Demis Hassabis served on FHI's advisory board. The 2015 Puerto Rico AI safety conference, the 2017 Asilomar Principles, and later the 2023 Bletchley Declaration at the UK AI Safety Summit all drew on a vocabulary of catastrophic and existential risk that FHI had helped to develop.

Fourth, FHI seeded successor organisations. The Centre for the Governance of AI (GovAI), founded by Dafoe within FHI in 2018, became an independent non-profit in 2021 after the opportunity arose for Dafoe to take on a senior role at DeepMind; the university would not allow him to hold a dual appointment.^[15][16] The Global Priorities Institute, founded at Oxford in 2018 by William MacAskill and Hilary Greaves, overlapped substantially with FHI's macrostrategy work and continued after FHI's closure. The Forethought Foundation for Global Priorities Research and the Macrostrategy Research Initiative also drew on FHI personnel.

How was the Future of Humanity Institute funded?

FHI was funded through a mix of sources that shifted across its lifetime. Initial support came from James Martin's gift to what became the Oxford Martin School.^[6][20] Later donors included the European Research Council, the Leverhulme Trust, the insurance company Amlin, the Future of Life Institute (a separate organisation co-founded by Max Tegmark and Jaan Tallinn that shares ideological roots but is institutionally distinct), Open Philanthropy (which recommended a 13.4 million pound grant in 2018), and Elon Musk's 1 million pound 2015 contribution to AI risk research.^[5] By the late 2010s a substantial share of the institute's funding came from donors associated with the effective altruism movement.

Funding was not the proximate cause of the closure. Sandberg's Final Report and contemporaneous reporting indicate that, at the time of the closure, FHI had committed funds and outside donors willing to provide more.^[1][4] The constraint was administrative rather than financial.

The 2023 Bostrom email controversy

In early January 2023, Bostrom posted on his personal website a document titled "Apology for an Old Email" in which he addressed an email he had sent in 1996 to the Extropians mailing list as a graduate student.^[13] The 1996 email contained the sentence "Blacks are more stupid than whites" and used a racial slur in describing how he expected the statement to be received. Bostrom said he had been alerted that an archive of the list was being searched and wanted to get ahead of the discovery. He described the original email as "disgusting" and said he "completely" repudiated it, though parts of his apology, including comments on intelligence and genetics, were widely criticised as inadequate.^[13]

The University of Oxford opened an investigation. The investigation concluded on 10 August 2023 with a statement that the university did "not consider you to be a racist or that you hold racist views" and that it considered Bostrom's January 2023 apology to have been sincere.^[14] The episode generated extensive coverage in The Daily Beast, The Oxford Student, The Oxford Blue, and elsewhere, and prompted debate within the effective altruism community about how to respond.

Reporting on the FHI closure repeatedly notes that the controversy contributed to administrative friction between the institute and the Faculty of Philosophy, although neither the university nor Bostrom has issued a definitive statement linking the two.^[4][5]

Why did the Future of Humanity Institute close?

On 16 April 2024 the Faculty of Philosophy at Oxford announced that the Future of Humanity Institute had been closed.^[1][4] The university's official statement said that after consideration of "the best structures for conducting our academic research" the decision had been made to close FHI, while recognising "the Institute's important contribution to this emerging field, which researchers elsewhere across the University are likely to continue".^[5]

The sequence of events leading to the closure, as described by Sandberg in his Final Report and in subsequent reporting, was as follows.^[1][6] Beginning in 2020, the Faculty of Philosophy imposed a freeze on hiring and on new fundraising at FHI. The reasons for that freeze were not made public. Senior researchers and junior staff began leaving as their contracts ended and could not be renewed. The Centre for the Governance of AI separated and became an independent non-profit in 2021. In late 2023, the Faculty informed the institute that the contracts of remaining FHI staff would not be renewed at all. Bostrom resigned from the University of Oxford in connection with the closure.^[5]

Bostrom characterised the situation as "a death by bureaucracy", explaining that "we were administratively housed within the faculty of philosophy, even though the majority of our research team by this time were non-philosophers".^[5] Sandberg used the phrase "gradual suffocation by Faculty bureaucracy".^[1] To illustrate the administrative load, FHI staff coined a joke measurement unit called "the Oxford", described in the Final Report and contemporaneous coverage "as a shorthand for the amount of work it takes to read and write 308 emails", which was the actual effort required to have a single small grant disbursed into the institute's account after both the funder and the university had already approved it.^[6] His Final Report, hosted on the Oxford University Research Archive, doubles as an oral history of the institute, an account of its scholarly contributions, and a set of lessons on running unconventional research groups inside traditional universities.^[1]

Successor and continuation organisations

FHI's intellectual programme did not end with the institute. Several organisations now carry forward overlapping research agendas.

Scroll sideways for more →

The Future of Life Institute, often confused with FHI, is a separate organisation founded in the United States in 2014 by Max Tegmark, Jaan Tallinn, Anthony Aguirre, Viktoriya Krakovna, and Meia Chita-Tegmark. FHI and FLI shared collaborators and intellectual concerns but were institutionally distinct.

Critiques and controversies

FHI was not without critics, and the criticism intensified as longtermism became more prominent in public debate. The main lines of critique are summarised below.

The most sustained external critic has been the philosopher Émile P. Torres, formerly Phil Torres, who once wrote in defence of longtermism but later argued that it is "quite possibly the most dangerous secular belief system in the world today".^[17] Writing in Salon in August 2022 and in subsequent academic and journalistic work, Torres argued that longtermism is rooted in transhumanist and eugenicist intellectual traditions and that its emphasis on enormous future populations risks justifying the sacrifice of present people for far-future projects.^[17] With the AI ethics researcher Timnit Gebru, Torres proposed the acronym TESCREAL (transhumanism, extropianism, singularitarianism, cosmism, rationalism, effective altruism, longtermism) to describe what they see as a related cluster of ideologies, several of which are tightly linked to FHI's intellectual circle.

Other critics have argued that the longtermist framework underweights near-term harms in favour of speculative far-future considerations, that it treats consequentialist calculations over very long horizons as more reliable than they actually are, and that the social composition of the field skews toward wealthy donors and a narrow demographic. Writers including Erik Hoel and the AI ethics researcher Kate Crawford have raised related concerns, although their critiques target the broader effective altruism and longtermism movements rather than FHI specifically. Some defenders within the field, including MacAskill, responded by reformulating longtermism in weaker forms that do not require the strongest assumptions about aggregating value across vast populations.

The 2023 Bostrom email controversy drew its own substantial body of criticism, both from outside commentators and from within the effective altruism community. Reactions on the Effective Altruism Forum and elsewhere were divided between those who accepted Bostrom's apology, those who found it inadequate, and those who argued that the entire episode reflected longstanding problems with the social culture around FHI and adjacent groups.

Legacy

FHI's institutional life was nineteen years. Its intellectual influence is likely to last considerably longer.

The modern field of AI safety, both in its technical and governance forms, is in large part the product of work that originated at or passed through FHI. Researchers trained or affiliated with the institute now hold senior positions at AI laboratories including Anthropic, OpenAI, and Google DeepMind, and at government bodies including the UK AI Safety Institute, the US AI Safety Institute, and the EU AI Office. The vocabulary of existential risk, AI alignment, information hazards, the unilateralist's curse, the vulnerable world hypothesis, and longtermism is in everyday use in policy and research discussions that did not exist in their current form before FHI began publishing.

Superintelligence remains foundational reading in the field a decade after publication, and Ord's The Precipice has played a similar role for the broader existential risk literature.^[10][12] Sandberg's Final Report, posted on the Oxford University Research Archive in April 2024, has been read both as an obituary for the institute and as a manual for future research groups attempting work that does not fit neatly inside traditional university structures.^[1]

The closure of FHI also raised harder questions about whether the modern university is well suited to host long-horizon, interdisciplinary, agenda-setting research of the kind FHI tried to do. Several proposals for successor institutes outside Oxford, by former staff and outside observers, are now in early stages.

See also

• Nick Bostrom
• Toby Ord
• Superintelligence
• Existential risk from AI
• AI safety
• AI Safety Institutes
• Center for AI Safety
• Effective Altruism
• William MacAskill

References

1. Sandberg, Anders. *Future of Humanity Institute 2005-2024: Final Report*. Oxford University Research Archive, April 2024. https://ora.ox.ac.uk/objects/uuid:8c1ab46a-061c-479d-b587-8909989e4f51 ↩
2. Future of Humanity Institute. Official site (memorial), https://www.futureofhumanityinstitute.org/ ↩
3. "Future of Humanity Institute". Wikipedia. https://en.wikipedia.org/wiki/Future_of_Humanity_Institute ↩
4. Weinberg, Justin. "The End of the Future of Humanity Institute". *Daily Nous*, 18 April 2024. https://dailynous.com/2024/04/18/end-future-of-humanity-institute/ ↩
5. "Oxford shuts down Elon Musk-funded Future of Humanity Institute". *The Oxford Student*, 20 April 2024. https://www.oxfordstudent.com/2024/04/20/oxford-shuts-down-elon-musk-funded-future-of-humanity-institute/ ↩
6. "Looking Back at the Future of Humanity Institute". *Asterisk Magazine*, Issue 8, 2024. https://asteriskmag.com/issues/08/looking-back-at-the-future-of-humanity-institute ↩
7. Bostrom, Nick. "Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards". *Journal of Evolution and Technology* 9, 2002. ↩
8. Bostrom, Nick. "Are You Living in a Computer Simulation?". *Philosophical Quarterly* 53(211), 2003.
9. Sandberg, Anders, and Nick Bostrom. *Whole Brain Emulation: A Roadmap*. FHI Technical Report 2008-3, 2008. https://www.fhi.ox.ac.uk/brain-emulation-roadmap-report.pdf
10. Bostrom, Nick. *Superintelligence: Paths, Dangers, Strategies*. Oxford University Press, 2014. ↩
11. Bostrom, Nick. "The Vulnerable World Hypothesis". *Global Policy* 10(4), 2019, pp. 455 to 476. https://nickbostrom.com/papers/vulnerable.pdf
12. Ord, Toby. *The Precipice: Existential Risk and the Future of Humanity*. Bloomsbury, 2020. ↩
13. Bostrom, Nick. "Apology for an Old Email". Personal site, January 2023. https://nickbostrom.com/oldemail.pdf ↩
14. "Nick Bostrom". Wikipedia. https://en.wikipedia.org/wiki/Nick_Bostrom ↩
15. Centre for the Governance of AI. "About Us". https://www.governance.ai/about-us ↩
16. "The Centre for the Governance of AI is becoming a nonprofit". EA Forum, 2021. https://forum.effectivealtruism.org/posts/zcAxoAHcSECyewr2t/the-centre-for-the-governance-of-ai-is-becoming-a-nonprofit ↩
17. Torres, Émile P. "Understanding 'longtermism': Why this suddenly influential philosophy is so toxic". *Salon*, 20 August 2022. https://www.salon.com/2022/08/20/understanding-longtermism-why-this-suddenly-influential-philosophy-is-so/ ↩
18. "Future of Humanity Institute 2005-2024: Final Report". Effective Altruism Forum mirror, April 2024. https://forum.effectivealtruism.org/posts/uK27pds7J36asqJPt/future-of-humanity-institute-2005-2024-final-report
19. "FHI (Future of Humanity Institute) has shut down (2005-2024)". LessWrong, April 2024. https://www.lesswrong.com/posts/tu3CH22nFLLKouMKw/fhi-future-of-humanity-institute-has-shut-down-2005-2024
20. Oxford Martin School. "1 million pound grant for research into long-term future". https://www.oxfordmartin.ox.ac.uk/news/201507-fli ↩
21. "Superintelligence: Paths, Dangers, Strategies". Wikipedia. https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies ↩
22. "The Precipice: Existential Risk and the Future of Humanity". Wikipedia. https://en.wikipedia.org/wiki/The_Precipice:_Existential_Risk_and_the_Future_of_Humanity ↩