Goodbye to FHI

The lights are out at the Future of Humanity Institute. Anders Sandberg has put together a ‘final report’ on FHI, some oral history on how FHI grew up and what it was like to be there. There is also a tombstone website with a short history and some greatest hits, publication-wise. Do read those if you have the time(sidenote: I’m largely stealing from them here anyway.). But since I was fortunate enough to spend a couple years there, and to get to know many FHI people, I thought to add to the pile of commentary.

There are things to be said about what exactly led to FHI closing; but mostly not here. This’ll just be a note of personal appreciation, and some sadness.

Founding

The most surprising thing is that the institute ever started.

If it started in one piece of writing, why not pin it to Nick Bostrom’s 1997 article ‘Predictions from Philosophy?’. Progress on important (especially technological) dimensions is both fast and exponential: “scientific knowledge [is] doubling every 10 to 20 years since the second world war”, and “computer processor speed doubling every 18 months or so”. And this is not a familiar predicament to be in. If we want to look much further out into the future, we might need a grab-bag of predictive tools: some from science, mixed and combined as needed, and perhaps some of the more speculative and general-purpose tools from philosophy. So Bostrom asks whether there is room for “a philosophy whose aim is prediction”, with a role for “the generalised scientist” to try making progress on questions that fall outside of narrower disciplines to otherwise be “consigned to […] the popular press, or just ignored”.

There was also a fortuitous meeting, a year before. Here is a 1996 (!) blog excerpt(sidenote: Archive link) from the philosopher David Pearce:

I spend a most agreeable afternoon wandering around Kew Gardens in an old-fashioned organic VR. My companion and intellectual conscience is the polymathic Swedish-born Niklas. […] Most academics’ idea of constructive criticism stretches little further than the accusation they are too modest about their own abilities. Niklas, on the other hand, will both hear and deliver criticism in complete equanimity. […] He should be meeting Anders in Stockholm next month. The cultural anthropologist in me eagerly awaits reports.

That is the same Anders Sandberg who would join FHI as it was established at the University of Oxford by Bostrom in 2005, originally intended as a 3-year project.

Here’s my impression of the common thread of motivations that largely survived from founding onwards:

1. To study extremely zoomed-out questions of decision-relevance: what do we see for humanity’s prospects when we look out from our vantage point in space and history? How might the world steer itself onto more hopeful trajectories?
2. To try to do this research well, meaning to aim for truth and importance and practical relevance. And turning that “what should we prioritise” lens onto the work itself, being prepared to drop research directions and set out on new ones, as the territory comes into clearer view(sidenote: See Hamming’s famous line of questioning: “What are the important problems of your field?”, “What important problems are you working on?”, and “If what you are doing is not important, and if you don’t think it is going to lead to something important, why are you at Bell Labs working on it?”). Academic prestige might proxy for this, but it shouldn’t become the end goal.

This meant being prepared to do research which seem a bit fringe, or avant-garde, or frankly just weird — treading into territory normally associated with cranks and (for some reason) retired physicists. So the idea was less about advancing the far frontiers of already recognised research (as important as that is), but:

to find things deserving of being recognized, show that they matter, invent the theoretical and conceptual tools needed to start to do useful work on them—and then (hopefully) hand it off to others as the topic matures.

That’s from Anders’ final report again.

Some highlights

Things got started quickly in Oxford, largely thanks to the fiscal generosity of the late Dr James Martin (who gave his name to the Oxford Martin School). Early research considered whole brain emulation, the likelihood of global catastrophe, the concept of a ‘singleton’.

Academic discussion soon spilled over to the public square with the 2006 launch of a collaborative blog called ‘Overcoming Bias’. Three years later, the scope of topics had inflated to require a new home again. Eliezer Yudkowsky, contributor to Overcoming Bias and longtime collaborator with FHI researchers, moved to a new community blog called ‘LessWrong’.

In 2009, Nick Bostrom began work on a book on existential risks. One of the chapters kept growing until it was obvious its topic could consume a book of its own. This was the chapter on AI; the book became 2014’s Superintelligence: Paths, Dangers, Strategies. In the intervening years, papers were written and hires were made on AI safety. FHI had stepped into a longstanding thread of thinking around intelligent machines(sidenote: Indeed a thread Bostrom had contributed to already, as in Bostrom (1997). A note from his website: “I’ve now been alive long enough to have seen a significant shift in attitudes to these questions. Back in the 90s, they were generally regarded as discreditable futurism or science fiction - certainly within academia. They were left to a small set of “people on the Internet”, who were at that time starting to think through the implications of future advances in AI and other technologies, and what these might mean for human society.” For ‘people on the Internet’, see (for an influential if profligate example) the Extropians mailing list. For much earlier examples, see Turing (1950) or Good (1965) and (1970). This page has many more examples.), and became one of the most significant institutions for developing and popularising ideas around AI risk — ideas which have recently become far better known.

In 2017, FHI launched a ‘Governance of AI Program’ — led by Allan Dafoe — to study questions around policy and advanced AI, beyond the narrowly technical questions. As far as I understand, it was the first serious research effort on what’s called ‘AI governance’,(sidenote: Specifically governance questions focused on making sure transformative AI goes well — the kind of AI that doesn’t quite exist but might soon. There was and remains much work on policy questions around software, algorithms, and present-day AI.) now a burgeoning field. In 2021, that research group span out of FHI to become an independent think tank, the Centre for the Governance of AI.

From roughly 2018 onwards came a research group focused on biological risks. It made the case for biological risk reduction as a global priority, specifically because of engineered pandemics. Like with AI risk, this work was hardly the first to suggest that pandemics are bad, or that humanity might consider doing more to try preventing major pandemics. But it went further than most previous work in mounting a quantitative and systematic case for taking pandemic risk far more seriously. Shortly after, this team became very busy.

The very onset of Covid — early 2020 — also saw the release of Toby Ord’s The Precipice. The topic is ‘existential risks’ — risks that threatens the destruction of humanity’s longterm potential. There are chapters on unaligned AI and engineered pandemics, and more besides, but that organising theme explains what connects them as objects of special concern.

An even more recent development was two new hires and the formation of a research group working on ethical and strategic issues around digital minds. They helped organise a major multi-author report on ‘Insights from the Science of Consciousness’.

Attitudes and impact

The writer Tim Urban sometimes talks about ‘Idea Labs’, a kind of archetype for a research institution. From What’s Our Problem:

People in an Idea Lab see one another as experimenters and their ideas as experiments […] unearned conviction is a major no-no in an Idea Lab. So someone with a reputation for bias or arrogance or dishonesty will be met with a high degree of skepticism, no matter how much conviction they express.

From my impression of briefly being part of FHI (from 2020–2022), and from hearing about the earlier years, I think this gets at something FHI did well — this combination of (i) unusual openness and tolerance for ‘weird’ ideas, but (ii) unusual seriousness about getting them right.

This thoughtful comment mentions some more specific factors that made FHI distinctive. There was a lot of effort to cut through bullshit in group discussions — especially important when misguided ideas can survive and grow without the natural predator of feedback from empirical tests(sidenote: This is an especially thorny problem when you’d really prefer not to learn from experience. Carl Sagan (1983): “Theories that involve the end of the world are not amenable to experimental verification—or at least, not more than once”). And there was a vigilance about maintaining high research standards in hiring and visitors.

There have always been venues to discuss very big-picture topics like “what might end the world”, “are we in a simulation”, or “could we lose control of AI”. These include internet forums, smoke-filled dorm rooms, flyers handed out on the street, and (more recently) bro-coded podcast interviews. The scarce thing was to take these questions as seriously as any other technical or philosophical question, and conviction in the first place that it is possible to do much better than dorm-room discourse.

So earnestness and high standards were one thing. But that does not mean the many seminars and ad-hoc whiteboard discussions were not also, for a certain kind of technically and philosophically inclined nerd, unmissably fun. In terms of the breadth and interestingness of water cooler conversations, FHI was top-tier. Here’s a small but representative sample of informal seminars I took notes from: on refining the concept of existential hope; on ethical considerations around directed panspermia; on whether the Kelly criterion is a useful guide for humanity’s ideal risk tolerance (no); on what’s wrong with the doomsday argument; on which kinds of uncertainty count as good reasons for discounting the future; on the space of possible reforms to scientific institutions.

Also I think there are lessons about how to do inderdisciplinary(sidenote: Transdisciplinary? Multidisciplinary? Cross-disciplinary?) work productively(sidenote: Which isn’t saying that FHI’s outputs and perspectives were diverse on many important dimensions, they weren’t. There was a house style to FHI’s technical reports which was, well, technical and somewhat nerdy. And shorter thrift was given to nuanced social and political questions over questions where more confident claims or predictions looked feasible. Of course such a broad remit invites such “why don’t you work on X” questions.). Being ‘interdisciplinary’ is a selling point on grant applications, but sometimes has the feeling (to me) of a summit between delegates from two countries: lots of shaking hands and shows of mutual understanding and commonalities and gains from trade, but still the delegates go home to their own countries. I think FHI was less self-conscious here: the walls of the departments just mattered less simpliciter. Plus, there was a lot of openness to finding insights which had fallen somewhere in the cracks between zones of intense research pressure, or far behind any specific (sidenote: A nice example here is Toby Ord’s paper on ‘The Edges of Our Universe’ or his poster on the various boundaries of a black hole. The concepts and maths in both cases are high-school level, if a notch above popular science nonfiction book level. So the incentives to write it are somehow rare both inside and out of the relevant academic fields, viz. cosmology or astronomy.).

I really think it is worth appreciating the number and depth of insights that FHI can claim significant credit for(sidenote: Again, many of these ideas or close precursors were already in the water in some form, and in those cases the work was to majorly clarify or extend them. But isn’t that true of most broadly significant philosophical ideas?). In no particular order:

• The concept of existential risk, and arguments for treating x-risk reduction as a global priority (see: The Precipice)
• Arguments for x-risk from AI, and other philosophical considerations around superintelligent AI (see: Superintelligence)
• Arguments for the scope and importance of humanity’s long-term future (since called longtermism)
• Information hazards
• Observer selection effects and ‘anthropic shadow’
• Bounding natural extinction rates with statistical methods
• The vulnerable world hypothesis
• Moral trade
• The moral imperative towards cost-effectiveness in global health
• Crucial considerations
• The unilteralist’s curse
• Dissolving the Fermi paradox
• The reversal test in applied ethics
• 'Comprehensive AI services’ as an alternative to unipolar outcomes
• The concept of existential hope

Note how much of the literal terminology was coined on (one imagines) a whiteboard in FHI. “Existential risk” isn’t a neologism, but I understand it was Nick who first suggested it be used in a principled way to point to the “loss of potential” thing. “Existential hope”, “vulnerable world”, “unilateralist’s curse”, “information hazard”, all (as far as I know) tracing back to an FHI publication.

It’s also worth restating on the areas of study that FHI effectively incubated, and which are now full-blown fields of research. It was early on technical research around AI risk, very early on AI governance research, early on getting more strategic clarity around biosecurity. And if research on digital minds and their implications grows to become something resembling a ‘field’, then the small team and working groups on digital minds can make a claim to precedence, as well as early and more recent published work. Especially for its size, FHI was staggeringly influential.

Headwinds and lessons

Like I said, I don’t think this is the place for a play-by-play account of what caused FHI to close. The two main reasons are that (i) I’m not a spokesperson for FHI, so it would just be inappropriate; (ii) I don’t know many details; and (iii) I don’t expect particular details to change the impression you’d get from reading the ‘final report’.

Another reason is that shortly after someone dies, it just feels most appropriate to trade appreciative stories of that person’s life as a whole, not to launch into the grizzly details of their end-of-life health complications, or to tally up all the stupid missteps that person ever made.

That said, my personal sense is that contributing mistakes were made, in the sense that things could have gone better (for everyone) if the tape were replayed.

In particular, the ops situation was dysfunctional, in a way that might have been fixed while there was still a chance. In any case it seems worth strongly emphasising the work of the ops people to keep FHI running as long as it did, especially in the later years where the ratchet mechanism of a hiring freeze meant ops staff could leave but not be replaced. That ops work was largely invisible and interfacing with the bureaucracy was, by all accounts, suffocating. See this comment, it is worth reading.

To be clear, I don’t think this happened because FHI researchers themselves were unusually demanding. Rather, the rationale was to build a kind of sheltered garden around the researchers, free as far as possible from hostile outside forces — and it fell almost entirely on the ops staff to provide that shelter.

Still, Carrick points out an tension in how to relate to all this: on one hand, the operations staff bore an inhuman burden; this should be widely known and thanked. On the other, this was not an overall state of affairs to encourage, or glorify, or repeat. Even success would have been Pyrrich.

Anders puts it like this:

I often described Oxford like a coral reef of calcified institutions built on top of each other, a hard structure that had emerged organically and haphazardly and hence had many little nooks and crannies where colorful fish could hide and thrive. FHI was one such fish but grew too big for its hole.

As for lessons, who knows? Something about what progress can be made in a small place in a relatively short span of time by very committed misfits relatively unconcerned about academic prestige. Something hackneyed like that.

Should there be another FHI? I’m not sure. Compared to the world in 2005, the attitude and subjects of FHI’s research are in safer hands. Governments are forming entire new departments around some of the catastrophic risks few were theorising about then. There are many, many new research and advocacy initiatives. And alumni are doing amazing things — heading major think tanks, leading safety research at AI labs, advising governments. If FHI were a candle then it’s since lit many flames, and the matchbox might not be needed again.