A few reasons for x-risk scepticism

Recently I stumbled on this post, arguing ‘in favour of exploring nagging doubts about x-risk(sidenote: Meaning “existential risk”; risks that threaten the destruction of humanity’s longterm potential. “Doubts about existential risk” meaning doubts about its usefulness as a concept, the importance of prioritising them, the plausibility of proposed sources of x-risk.)’ —

I usually feel good about people in this situation paying attention to their gut scepticism or nagging doubts about x-risk […] partly because it seems like it would be a public good to explore and write about these things. Either their gut is onto something with parts of its scepticism, in which case it would be great to have that articulated; or their gut is wrong, but if other people have similar gut reactions then playing out that internal dialogue in public could be pretty helpful.^[1]

These dynamics seem especially strong in the case of AI risk — which I regard as the most serious source of x-risk, but also the one where I most wish people spent more time exploring their nagging doubts.

I agree with this(sidenote: “This” being the thought that (for the most part) proponents of bold action-guiding philosophies should openly share the reasons that make them hesitate the most.) and so I left some x-risk sceptical thoughts of my own as a comment. I’m reproducing what I wrote below, and may add to it in future.

1. You could reasonably think human extinction this century is very unlikely. One way to reach this conclusion is simply to work through the most plausible causes of human extinction, and reach low odds. Vasco Grilo does this for (great power) conflict and nuclear winter, John Halstead suggests extinction from climate change is very low here, and the background rate of extinction from natural sources can be bounded by (among other things) observing how long humans have already been around for. That leaves extinction risk from AI and (AI-enabled) engineered pandemics, where discussion is more scattered and inconclusive. Here and here are some reasons for scepticism about AI existential risk.
   • Even if the arguments for AI x-risk are sound, then it’s not clear how they are arguments for expecting literal human extinction over outcomes like ‘takeover’ or ‘disempowerment’. It’s hard to see why AI takeover would lead to smouldering ruins, versus continued activity and ‘life’, just a version not guided by humans, or by human values, or endorsed by most humans.
2. So “existential catastrophe” probably shouldn’t just mean “human extinction”. But then it is surprisingly slippery as a concept. Existential risk is the risk of existential catastrophe, but it’s difficult to give a neat and intuitive definition of “existential catastrophe” such that “minimise existential catastrophe” is a very strong guide for how to do good. Hilary Greaves dicusses candidate definitions here.
3. From (1), you might think that if x-risk reduction this century should be a near-top priority, then most its importance comes from mitigating non-extinction catastrophes, like irreversible dystopias. But few current efforts are explicitly framed as ways to avoiding dystopian outcomes and it’s less clear how to do that, other than preventing AI disempowerment or takeover, assuming that is dystopian.
4. But then isn’t x-risk work basically just about AI, and maybe also biorisk? Shouldn’t specific arguments for those risks and ways to prevent them therefore matter more than more abstract arguments for the value of mitigating existential risks in general?
5. Many strategies to mitigate x-risks trade off uncomfortably against other goods. Of course they require money and talent, but it’s hard to argue the world is spending too much on e.g. preventing engineered pandemics. But (to give a random example), mitigating x-risk from AI might require strong AI control measures. If we also end up thinking things like AI autonomy matter, that could be an uncomfortable (if worthwhile) price to pay(sidenote: To put it more bluntly: there might be cases where the only way to keep humans ‘on top’ is to do something which resembles enslaving AI systems, “prevent[ing] them from forming separate cultures they control with distinct norms and values”).
6. It’s not obvious that efforts to improve prospects for the long-run future should focus on preventing unrecoverable disasters. There is a strong preemptive argument for this; roughly that humans are likely to recover from less severe disasters, and so retain most their prospects (minus the cost of recovering, which is assumed to be small in terms of humanity’s entire future). The picture here is one on which the value of the future is roughly bimodal — either we mess up irrecoverable and achieve close to zero of our potential, or we reach roughly our full potential. But that bimodal picture isn’t obviously true. It might be comparably important to find ways to turn a mediocre-by-default future into a really great future, for instance.
7. The case for prioritising x-risk mitigation often involves mistakes, like strong ‘time of perils’ assumptions and apples to oranges comparisons. A naive case for prioritising x-risk mitigation might go like this: “reducing x-risk this century by 1 percentage point is worth one percentage point of the expected value of the entire future conditional on no existential catastrophes. And the entire future is huge, it’s like $10^x$ lives. So reducing x-risk by even a tiny fraction, say $y\%$, this century saves $10^{(x-2)\cdot y}$ (a huge number of) lives in expectation. The same resources going to any work directed at saving lives within this century cannot save such a huge number of lives in expectation even if it saved 10 billion people.” This is too naive for a couple reasons:
   • This assumes this century is the only time where an existential catastrophe could occur. Better would be “the expected value of the entire future conditional on no existential catastrophe this century”, which could be much lower.
   • This compares long-run effects with short-run effects without attempting to evaluate the long-run effects of interventions not deliberately targeted at reducing existential catastrophe this century.
8. Naive analysis of the value of reducing existential catastrophe also doesn’t account for ‘which world gets saved’. This feels especially relevant when assessing the value of preventing human extinction, where you might expect the worlds where extinction-preventing interventions succeed in preventing extinction are far less valuable than the expected value of the world conditional on no extinction (since narrowly avoiding extinction is bad news about the value of the rest of the future). Vasco Grilo explores this line of thinking here, and I suggest some extra thoughts here.
9. The fact that some existential problems (e.g. AI alignment) seem, on our best guess, just about solvable with an extra push from x-risk motivated people doesn’t itself say much about the chance that x-risk motivated people make the difference in solving those problems (if we’re very uncertain about how difficult the problems are). Here are some thoughts about that.

These thoughts make me hesitant about confidently thinking and acting as if x-risk is overwhelmingly important, even compared to other potential ways to improve the long-run future, or other framings on the importance of helping navigate the transition to very powerful AI.

But I also think that introducing and refining “existential risk” marked a real and rare kind of breakthrough in big-picture moral thinking. And I like this snippet from the FAQ page for The Precipice —

But for most purposes there is no need to debate which of these noble tasks is the most important—the key point is just that safeguarding humanity’s longterm potential is up there among the very most important priorities of our time.