Worrying facts from 'The Precipice'

I recently finished reading Toby Ord’s new book ‘The Precipice: Existential Risk and the Future of Humanity’. The book discusses the risks of catastrophic events that destroy all or nearly all of humanity’s potential. There are many of them, including but not limited to the Hollywood scenarios that enter most people’s minds: asteroids, supervolcanoes, pandemics (natural and human-engineered), dystopian political ‘lock-in’, runaway climate scenarios, and unaligned artificial general intelligence. The overall risk of an existential catastrophe this century? Roughly one in six, this author guesses: Russian roulette. Clearly, mitigating existential risk is not nearly treated like the overwhelmingly important global priority it is: not in our political institutions, nor in popular consciousness. Anyway, it’s excellent– highly recommended.

It was also full of some fairly alarming and/or surprising facts. So in place of a full review, here are some highlights:

• The total amount of money expended annually on researching and mitigating existential risks is dwarfed by the amount of money spent annually on ice-cream, by about two orders of magnitude.
• The Biological Weapons Convention is the international body responsible for the continued prohibition of bioweapons, which on Toby’s estimate pose a greater existential risk by an order of magnitude than the combined risk from nuclear war, runaway climate change, asteroids, supervolcanoes, and naturally arising pandemics. Its annual budget is less than that of the average MacDonald’s restaurant. (p.57)
• Remember that quotation attributed to Einstein that “If the bee disappeared off the surface of the globe then man would only have four years of life left”? Firstly, it’s not true– a recent review found that the loss of all pollinators would create a 3 to 8 percent reduction in global crop production. Secondly, Einstein never said it. (p.118)
• Technological progress is really hard to predict – “One night in 1933, the world’s pre-eminent expert on atomic science, Ernest Rutherford, declared the idea of harnessing atomic energy to be ‘moonshine’. And the very next morning Leo Szilard discovered the idea of the chain reaction. In 1939, Enrico Fermi told Szilard the chain reaction was but a ‘remote possibility’, and four years later Fermi was personally overseeing the world’s first nuclear reactor.” Furthermore, at the start of the 20th century, many thought heavier-than-air human flight to be impossible. Wilbur Wright was somewhat more optimistic, guessing it to be at least 50 years away; 2 years before he invented it. (p.121)
• The UK has four levels of ‘biosafety’. The highest level, ‘BSL-4’, is reserved for research involving the most dangerous and infectious pathogens. The 2001 outbreak of foot-and-mouth disease caused economic damages totaling £8 billion and the slaughter of some 6 million animals to halt its spread. Six years later, a lab was researching the disease under BSL-4 security. Another outbreak that year was traced back to a leaky pipe in the lab, spreading the disease into the groundwater. “After an investigation, the lab’s license was renewed—only for another leak to occur two weeks later.” (p.130)
• Nuclear near misses have been terrifyingly frequent. The book lists more than ten examples. Here are a few:
  • 1958 “A B-47 bomber accidentally dropped a nuclear bomb over South Carolina, landing in someone’s garden and destroying their house.” (the warhead remained in the plane)
  • 27th October 1962. Four nuclear submarines had been sent by the Soviet Union to support their military operations in Cuba during the height of the Missile Crisis. A US warship detected one of these submarines and tried to force it to surface by using depth charges as ‘warning shots’. The submarine had been underwater for days and had lost radio contact for as long– and with it, information about the situation unfolding above. Moreover, being designed for the Arctic, the submarine was breaking down in the tropical waters. Temperatures ranged from 45°C to 60°C as carbon dioxide began to accumulate. Crew members were falling unconscious. The captain, Valentin Savitsky, guessed by the bombardment that war had broken out. He ordered his crew to prepare the submarine’s nuclear weapon. Firing this required the agreement of another officer who held the other half of the firing key. He consented. “On any of the other submarines, this would have sufficed to launch their nuclear weapon. But by the purest luck, submarine B-59 carried the commander of the entire flotilla… [who] refused to grant it. Instead, he talked Captain Savitsky down from his rage.” (p.4)
  • 28th October 1962 The very next day, a US base in and US-occupied Japanese island received by radio an order to launch its nuclear arsenal. “All three parts of the coded order matched the base’s own codes, confirming that it was a genuine order to launch their nuclear weapons.” Captain William Bassett took command and became responsible for executing the order. But he grew suspicious– a pre-emptive strike should already have hit them, and the threat level was set to DEFCON 2 rather than the highest level of DEFCON 1. Yet, he radioed the Missile Operations Centre to check, and received very same order. A lieutenant in charge of a different launch site told Bassett he had no right to stop the launch given the order was repeated. “In response, Bassett ordered two airmen from an adjacent launch site to run through the underground tunnel to the site where the missiles were being launched, with orders to shoot the lieutenant if he continued without either Bassett’s agreement or a decleration of DEFCON 1”. Bassett then called the Missile Operations Centre again, who this time issued an order to stand down. This story is still disputed and was only made public in 2015.
  • 26 September 1983. Just after midnight, the Soviet early-warning system designed to indicate nuclear launches from the United States showed five ICBMs heading towards Russia. The duty officer, Stanislav Petrov, was under orders to report such a warning to his superiors, who in turn were instructed to retaliate in kind with immediate effect. “For five tense minutes he considered the case, then despite his remaining uncertainty, reported it to his commanders as a false alarm.” (p.96)
• Norman Borlaug was an American agronomist who developed new high-yield and disease-resistant varieties of wheat during the so-called ‘Green Revolution’. He may have saved more lives than any person who ever lived. Estimates range from 260 million to over a billion lives saved. (p.97)

Not so much a fact, but an interesting thought: consider a toy model of existential risk where each century is exposed to an equal amount of risk $r$ (a constant hazard rate), and each century prior to the catastrophe has some constant value $v$. Then the expected value of the future would be: $EV=\sum_{i=0}^{\infty}(1-r)^i v=\frac{v}{r}$. So the expected disvalue of this century’s existential risk is $r\frac{v}{r}=v$. Notice, perhaps surprisingly, how it follows that the value of reducing this century’s existential risk by some proportion is independent of the initial risk, of the hazard rate. This does also make sense intuitively: the higher the risk each century (the hazard rate), the shorter the expected length of our future; so the stakes are lower while lowering this century’s absolute risk is ‘easier’. On the other hand, the lower the hazard rate, the longer and larger our expected future; so the stakes are higher while the risk this century ‘harder’ to reduce. The value of halving all future risk is $\frac{v}{r/2}-\frac{v}{r}=\frac{v}{r}$. More generally, the value of reducing all future risk by some proportion $x$ is equal to $\frac{xv}{r(1-x)}$. Notice that the value of reducing risk over all centuries by some fixed proportion is higher the lower the risk per century (hazard rate). Further:

Suppose you thought it equally likely the value of reducing risk was ten times as important as the basic model suggests, or a tenth as important. The average of these is not one times as important: it is 5.05 times as important.

Again, strongly recommended; but not at all comforting.

The Precipice and the Land Beyond - Hilary Paynter

More resources:

• Toby’s interview on the 80,000 Hours podcast.
• The audiobook, narrated by the author.
• Toby’s recent talk and Q&A at the Effective Altruism Global Conference
• Toby’s talk at the Cambridge Union
• existential-risk.org website
  • Nick Bostrom’s ‘Existential Risk Prevention as Global Priority’