Existential Risk and the Future of Humanity

The world is in the early stages of what may be the most deadly pandemic of the past 100 years. In China, thousands of people have already died; large outbreaks have begun in South Korea, Iran and Italy; and the rest of the world is bracing for impact. We do not yet know whether the final toll will be measured in thousands or hundreds of thousands. For all our advances in medicine, humanity remains much more vulnerable to pandemics than we would like to believe.

To understand our vulnerability, and to determine what steps must be taken to end it, it is useful to ask about the very worst-case scenarios. Just how bad could a pandemic be? In science fiction, we sometimes encounter the idea of a pandemic so severe that it could cause the end of civilisation, or even of humanity itself. Such a risk to humanity’s entire future is known as an existential risk. We can say with certainty that the novel coronavirus, named Covid-19, does not pose such a risk. But could the next pandemic? To find out, and to put the current outbreak into greater context, let us turn to the past.

In 1347, death came to Europe. It entered through the Crimean town of Caffa, brought by the besieging Mongol army. Fleeing merchants unwittingly carried it back to Italy. From there, it spread to France, Spain and England. Then up as far as Norway and across the rest of Europe – all the way to Moscow. Within six years, the Black Death had taken the continent.

Tens of millions fell gravely ill, their bodies succumbing to the disease in different ways. Some bore swollen buboes on their necks, armpits and thighs; some had their flesh turn black from haemorrhaging beneath the skin; some coughed blood from the necrotic inflammation of their throats and lungs. All forms involved fever, exhaustion and an intolerable stench from the material that exuded from the body.

There were so many dead that mass graves needed to be dug and, even then, cemeteries ran out of room for the bodies. The Black Death devastated Europe. In those six years, between a quarter and half of all Europeans were killed. The Middle East was ravaged, too, with the plague killing about one in three Egyptians and Syrians. And it may have also laid waste to parts of central Asia, India and China. Due to the scant records of the 14th century, we will never know the true toll, but our best estimates are that somewhere between 5% and 14% of all the world’s people were killed, in what may have been the greatest catastrophe humanity has seen.

The Black Death was not the only biological disaster to scar human history. It was not even the only great bubonic plague. In AD541 the plague of Justinian struck the Byzantine empire. Over three years, it took the lives of roughly 3% of the world’s people.

When Europeans reached the Americas in 1492, the two populations exposed each other to completely novel diseases. Over thousands of years, each population had built up resistance to their own set of diseases, but were extremely susceptible to the others. The American peoples got by far the worse end of the exchange, through diseases such as measles, influenza and, especially, smallpox.

During the next 100 years, a combination of invasion and disease took an immense toll – one whose scale may never be known, due to great uncertainty about the size of the pre-existing population. We can’t rule out the loss of more than 90% of the population of the Americas during that century, though the number could also be much lower. And it is very difficult to tease out how much of this should be attributed to war and occupation, rather than disease. At a rough estimate, as many as 10% of the world’s people may have been killed.

Centuries later, the world had become so interconnected that a truly global pandemic was possible. Towards the end of the first world war, a devastating strain of influenza, known as the 1918 flu or Spanish flu, spread to six continents, and even remote Pacific islands. About a third of the world’s population were infected and between 3% and 6% were killed. This death toll outstripped that of the first world war.

Yet even events like these fall short of being a threat to humanity’s long-term potential. In the great bubonic plagues we saw civilisation in the affected areas falter, but recover. The regional 25%-50% death rate was not enough to precipitate a continent-wide collapse. It changed the relative fortunes of empires, and may have substantially altered the course of history, but if anything, it gives us reason to believe that human civilisation is likely to make it through future events with similar death rates, even if they were global in scale.

The Spanish flu pandemic was remarkable in having very little apparent effect on the world’s development, despite its global reach. It looks as if it was lost in the wake of the first world war, which, despite a smaller death toll, seems to have had a much larger effect on the course of history.

The full history of humanity covers at least 200,000 years. While we have scarce records for most of these 2,000 centuries, there is a key lesson we can draw from the sheer length of our past. The chance of human extinction from natural catastrophes of any kind must have been very low for most of this time – or we would not have made it so far. But could these risks have changed? Might the past provide false comfort?

Our population now is a thousand times greater than it was for most of human history, so there are vastly more opportunities for new human diseases to originate. And our farming practices have created vast numbers of animals living in unhealthy conditions within close proximity to humans. This increases the risk, as many major diseases originate in animals before crossing over to humans. Examples include HIV (chimpanzees), Ebola (bats), Sars (probably civets or bats) and influenza (usually pigs or birds). We do not yet know where Covid-19 came from, though it is very similar to coronaviruses found in bats and pangolins. Evidence suggests that diseases are crossing over into human populations from animals at an increasing rate.

Modern civilisation may also make it much easier for a pandemic to spread. The higher density of people living together in cities increases the number of people each of us may infect. Rapid long-distance transport greatly increases the distance pathogens can spread, reducing the degrees of separation between any two people. Moreover, we are no longer divided into isolated populations as we were for most of the past 10,000 years.

Together these effects suggest that we might expect more new pandemics, for them to spread more quickly, and to reach a higher percentage of the world’s people.

But we have also changed the world in ways that offer protection. We have a healthier population; improved sanitation and hygiene; preventative and curative medicine; and a scientific understanding of disease. Perhaps most importantly, we have public health bodies to facilitate global communication and coordination in the face of new outbreaks. We have seen the benefits of this protection through the dramatic decline of endemic infectious disease over the past century (though we can’t be sure pandemics will obey the same trend). Finally, we have spread to a range of locations and environments unprecedented for any mammalian species. This offers special protection from extinction events, because it requires the pathogen to be able to flourish in a vast range of environments and to reach exceptionally isolated populations such as uncontacted tribes, Antarctic researchers and nuclear submarine crews.

It is hard to know whether these combined effects have increased or decreased the existential risk from pandemics. This uncertainty is ultimately bad news: we were previously sitting on a powerful argument that the risk was tiny; now we are not.

We have seen the indirect ways that our actions aid and abet the origination and spread of pandemics. But what about cases where we have a much more direct hand in the process – where we deliberately use, improve or create the pathogens?

Our understanding and control of pathogens is very recent. Just 200 years ago, we didn’t even understand the basic cause of pandemics – a leading theory in the west claimed that disease was produced by a kind of gas. In just two centuries, we discovered it was caused by a diverse variety of microscopic agents and we worked out how to grow them in the lab, to breed them for different traits, to sequence their genomes, to implant new genes and to create entire functional viruses from their written code.

This progress is continuing at a rapid pace. The past 10 years have seen major qualitative breakthroughs, such as the use of the gene editing tool Crispr to efficiently insert new genetic sequences into a genome, and the use of gene drives to efficiently replace populations of natural organisms in the wild with genetically modified versions.

This progress in biotechnology seems unlikely to fizzle out anytime soon: there are no insurmountable challenges looming; no fundamental laws blocking further developments. But it would be optimistic to assume that this uncharted new terrain holds only familiar dangers.

To start with, let’s set aside the risks from malicious intent, and consider only the risks that can arise from well-intentioned research. Most scientific and medical research poses a negligible risk of harms at the scale we are considering. But there is a small fraction that uses live pathogens of kinds that are known to threaten global harm. These include the agents that cause the Spanish flu, smallpox, Sars and H5N1 or avian flu. And a small part of this research involves making strains of these pathogens that pose even more danger than the natural types, increasing their transmissibility, lethality or resistance to vaccination or treatment.

In 2012, a Dutch virologist, Ron Fouchier, published details of an experiment on the recent H5N1 strain of bird flu. This strain was extremely deadly, killing an estimated 60% of humans it infected – far beyond even the Spanish flu. Yet its inability to pass from human to human had so far prevented a pandemic. Fouchier wanted to find out whether (and how) H5N1 could naturally develop this ability. He passed the disease through a series of 10 ferrets, which are commonly used as a model for how influenza affects humans. By the time it passed to the final ferret, his strain of H5N1 had become directly transmissible between mammals.

The work caused fierce controversy. Much of this was focused on the information contained in his work. The US National Science Advisory Board for Biosecurity ruled that his paper had to be stripped of some of its technical details before publication, to limit the ability of bad actors to cause a pandemic. And the Dutch government claimed that the research broke EU law on exporting information useful for bioweapons. But it is not the possibility of misuse that concerns me here. Fouchier’s research provides a clear example of well-intentioned scientists enhancing the destructive capabilities of pathogens known to threaten global catastrophe.

Of course, such experiments are done in secure labs, with stringent safety standards. It is highly unlikely that in any particular case the enhanced pathogens would escape into the wild. But just how unlikely? Unfortunately, we don’t have good data, due to a lack of transparency about incident and escape rates. This prevents society from making well-informed decisions balancing the risks and benefits of this research, and it limits the ability of labs to learn from each other’s incidents.

Security for highly dangerous pathogens has been deeply flawed, and remains insufficient. In 2001, Britain was struck by a devastating outbreak of foot-and-mouth disease in livestock. Six million animals were killed in an attempt to halt its spread, and the economic damages totalled £8bn. Then, in 2007, there was another outbreak, which was traced to a lab working on the disease. Foot-and-mouth was considered a highest-category pathogen, and required the highest level of biosecurity. Yet the virus escaped from a badly maintained pipe, leaking into the groundwater at the facility. After an investigation, the lab’s licence was renewed – only for another leak to occur two weeks later.

In my view, this track record of escapes shows that even the highest biosafety level (BSL-4) is insufficient for working on pathogens that pose a risk of global pandemics on the scale of the Spanish flu or worse. Thirteen years since the last publicly acknowledged outbreak from a BSL-4 facility is not good enough. It doesn’t matter whether this is from insufficient standards, inspections, operations or penalties. What matters is the poor track record in the field, made worse by a lack of transparency and accountability. With current BSL-4 labs, an escape of a pandemic pathogen is only a matter of time.

One of the most exciting trends in biotechnology is its rapid democratisation – the speed at which cutting-edge techniques can be adopted by students and amateurs. When a new breakthrough is achieved, the pool of people with the talent, training, resources and patience to reproduce it rapidly expands: from a handful of the world’s top biologists, to people with PhDs in the field, to millions of people with undergraduate-level biology.

The Human Genome Project was the largest ever scientific collaboration in biology. It took 13 years and $500m to produce the full DNA sequence of the human genome. Just 15 years later, a genome can be sequenced for under $1,000, and within a single hour. The reverse process has become much easier, too: online DNA synthesis services allow anyone to upload a DNA sequence of their choice then have it constructed and shipped to their address. While still expensive, the price of synthesis has fallen by a factor of 1,000 in the past two decades, and continues to drop. The first ever uses of Crispr and gene drives were the biotechnology achievements of the decade. But within just two years, each of these technologies were used successfully by bright students participating in science competitions.

Such democratisation promises to fuel a boom of entrepreneurial biotechnology. But since biotechnology can be misused to lethal effect, democratisation also means proliferation. As the pool of people with access to a technique grows, so does the chance it contains someone with malign intent.

People with the motivation to wreak global destruction are mercifully rare. But they exist. Perhaps the best example is the Aum Shinrikyo cult in Japan, active between 1984 and 1995, which sought to bring about the destruction of humanity. It attracted several thousand members, including people with advanced skills in chemistry and biology. And it demonstrated that it was not mere misanthropic ideation. It launched multiple lethal attacks using VX gas and sarin gas, killing more than 20 people and injuring thousands. It attempted to weaponise anthrax, but did not succeed. What happens when the circle of people able to create a global pandemic becomes wide enough to include members of such a group? Or members of a terrorist organisation or rogue state that could try to build an omnicidal weapon for the purposes of extortion or deterrence?

The main candidate for biological existential risk in the coming decades thus stems from technology – particularly the risk of misuse by states or small groups. But this is not a case in which the world is blissfully unaware of the risks. Bertrand Russell wrote of the danger of extinction from biowarfare to Einstein in 1955. And, in 1969, the possibility was raised by the American Nobel laureate for medicine, Joshua Lederberg: “As a scientist I am profoundly concerned about the continued involvement of the United States and other nations in the development of biological warfare. This process puts the very future of human life on earth in serious peril.”

In response to such warnings, we have already begun national and international efforts to protect humanity. There is action through public health and international conventions, and self-regulation by biotechnology companies and the scientific community. Are they adequate?

National and international work in public health offers some protection from engineered pandemics, and its existing infrastructure could be adapted to better address them. Yet even for existing dangers this protection is uneven and under-provided.

Despite its importance, public health is underfunded worldwide, and poorer countries remain vulnerable to being overwhelmed by outbreaks. Biotechnology companies are working to limit the dark side of the democratisation of their field. For example, unrestricted DNA synthesis would help bad actors overcome a major hurdle in creating extremely deadly pathogens. It would allow them to get access to the DNA of controlled pathogens such as smallpox (whose genome is readily available online) and to create DNA with modifications to make the pathogen more dangerous. Therefore, many synthesis companies make voluntary efforts to manage this risk, screening their orders for dangerous sequences. But the screening methods are imperfect, and they only cover about 80% of orders. There is significant room for improving this process, and a strong case for making screening mandatory.

We might also look to the scientific community for careful management of biological risks. Many of the dangerous advances usable by states and small groups have come from open science. And we’ve seen that science produces substantial accident risk. The scientific community has tried to regulate its dangerous research, but with limited success. There are a variety of reasons why this is extremely hard, including difficulty in knowing where to draw the line, lack of central authorities to unify practice, a culture of openness and freedom to pursue whatever is of interest, and the rapid pace of science outpacing that of governance. It may be possible for the scientific community to overcome these challenges and provide strong management of global risks, but it would require a willingness to accept serious changes to its culture and governance – such as treating the security around biotechnology more like that around nuclear power. And the scientific community would need to find this willingness before catastrophe strikes.

Threats to humanity, and how we address them, define our time. The advent of nuclear weapons posed a real risk of human extinction in the 20th century. There is strong reason to believe the risk will be higher this century, and increasing with each century that technological progress continues. Because these anthropogenic risks outstrip all natural risks combined, they set the clock on how long humanity has left to pull back from the brink.

I am not claiming that extinction is the inevitable conclusion of scientific progress, or even the most likely outcome. What I am claiming is that there has been a robust trend towards increases in the power of humanity, which has reached a point where we pose a serious risk to our own existence. How we react to this risk is up to us. Nor am I arguing against technology. Technology has proved itself immensely valuable in improving the human condition.

The problem is not so much an excess of technology as a lack of wisdom. Carl Sagan put this especially well: “Many of the dangers we face indeed arise from science and technology – but, more fundamentally, because we have become powerful without becoming commensurately wise. The world-altering powers that technology has delivered into our hands now require a degree of consideration and foresight that has never before been asked of us.”

Because we cannot come back from extinction, we cannot wait until a threat strikes before acting – we must be proactive. And because gaining wisdom takes time, we need to start now.

I think that we are likely to make it through this period. Not because the challenges are small, but because we will rise to them. The very fact that these risks stem from human action shows us that human action can address them. Defeatism would be both unwarranted and counterproductive – a self-fulfilling prophecy. Instead, we must address these challenges head-on with clear and rigorous thinking, guided by a positive vision of the longterm future we are trying to protect.

This is an edited extract from The Precipice: Existential Risk and the Future of Humanity by Toby Ord, published by Bloomsbury.

Why we need worst-case thinking to prevent pandemics. By Toby Ord. The Guardian, March 6, 2020.

We live during the most important era of human history. In the twentieth century, we developed the means to destroy ourselves – without developing the moral framework to ensure we won't. This is the Precipice, and how we respond to it will be the most crucial decision of our time.

Oxford moral philosopher Toby Ord explores the risks to humanity's future, from the familiar man-made threats of climate change and nuclear war, to the potentially greater, more unfamiliar threats from engineered pandemics and advanced artificial intelligence. With clear and rigorous thinking, Ord calculates the various risk levels, and shows how our own time fits within the larger story of human history. We can say with certainty that the novel coronavirus does not pose such a risk. But could the next pandemic? And what can we do, in our present moment, to face the risks head on?

A major work that brings together the disciplines of physics, biology, earth and computer science, history, anthropology, statistics, international relations, political science and moral philosophy, The Precipice is a call for a new understanding of our age: a major reorientation in the way we see the world, our history, and the role we play in it.

Bloomsbury

Toby Ord on the precipice and humanity’s potential futures.

By Robert Wiblin, Arden Koehler and Keiran Harris  

This week Oxford academic and 80,000 Hours trustee Toby Ord released his new book The Precipice: Existential Risk and the Future of Humanity. It’s about how our long-term future could be better than almost anyone believes, but also how humanity’s recklessness is putting that future at grave risk, in Toby’s reckoning a 1 in 6 chance of being extinguished this century.

I loved the book and learned a great deal from it.

While preparing for this interview I copied out 87 facts that were surprising to me or seemed important. Here’s a sample of 16:

1.       The probability of a supervolcano causing a civilisation-threatening catastrophe in the next century is estimated to be 100x that of asteroids and comets combined.

2.       The Biological Weapons Convention — a global agreement to protect humanity — has just four employees, and a smaller budget than an average McDonald’s.

3.       In 2008 a ‘gamma ray burst’ reached Earth from another galaxy, 10 billion light years away. It was still bright enough to be visible to the naked eye. We aren’t sure what generates gamma ray bursts but one cause may be two neutron stars colliding.

4.       Before detonating the first nuclear weapon, scientists in the Manhattan Project feared that the high temperatures in the core, unprecedented for Earth, might be able to ignite the hydrogen in water. This would set off a self-sustaining reaction that would burn off the Earth’s oceans, killing all life above ground. They thought this was unlikely, but many atomic scientists feared their calculations could be missing something. As far as we know, the US President was never informed of this possibility, but similar risks were one reason Hitler stopped pursuing the Bomb.

5.       If we eventually burn all the fossil fuels we’re confident we can access, the leading Earth-system models suggest we’d experience 9–13°C of warming by 2300, an absolutely catastrophic increase.

6.       In 1939, the renowned nuclear scientist Enrico Fermi told colleagues that a nuclear chain reaction was but a ‘remote possibility’. Four years later Fermi himself was personally overseeing the world’s first nuclear reactor. Wilbur Wright predicted heavier-than-air flight was at least fifty years away — just two years before he himself invented it.

7.       The Japanese bioweapons programme in the Second World War — which included using bubonic plague against China — was directly inspired by an anti-bioweapons treaty. The reasoning ran that if Western powers felt the need to outlaw their use, these weapons must especially good to have.

8.       In the early 20th century the Spanish Flu killed 3-6% of the world’s population. In the 14th century the Black Death killed 25-50% of Europeans. But that’s not the worst pandemic to date: that’s the passage of European diseases to the Americans, which may have killed as much as 90% of the local population.

9.       A recent paper estimated that even if honeybees were completely lost — and all other pollinators too — this would only create a 3 to 8 percent reduction in global crop production.

10.   In 2007, foot-and-mouth disease, a high-risk pathogen that can only be studied in labs following the top level of biosecurity, escaped from a research facility leading to an outbreak in the UK. An investigation found that the virus had escaped from a badly-maintained pipe. After repairs, the lab’s licence was renewed — only for another leak to occur two weeks later.

11.   Toby estimates that ‘great power wars effectively pose more than a percentage point of existential risk over the next century. This makes it a much larger contributor to total existential risk than all the natural risks like asteroids and volcanos combined.

12.   During the Cuban Missile Crisis, Kennedy and Khrushchev found it so hard to communicate, and the long delays so dangerous, that they established the ‘red telephone’ system so they could write to one another directly, and better avoid future crises coming so close to the brink.

13.   A US Airman claims that during a nuclear false alarm in 1962 that he himself witnessed, two airmen from one launch site were ordered to run through the underground tunnel to the launch site of another missile, with orders to shoot a lieutenant if he continued to refuse to abort the launch of his missile.

14.   In 2014 GlaxoSmithKline accidentally released 45 litres of concentrated polio virus into a river in Belgium. In 2004, SARS escaped from the National Institute of Virology in Beijing. In 2005 at the University of Medicine and Dentistry in New Jersey, three mice infected with bubonic plague went missing from the lab and were never found.

15.   The Soviet Union covered 22 million square kilometres, 16% of the world’s land area. At its height, during the reign of Genghis Khan’s grandson, Kublai Khan, the Mongol Empire had a population of 100 million, around 25% of world’s population at the time.

16.   All the methods we’ve come up with for deflecting asteroids wouldn’t work on one big enough to cause human extinction.

The 80,000 Hours Podcast.  Published March 7th, 2020

Many people have recently found that they want to read books offering the grandest perspectives possible on human existence, such as Sapiens by Yuval Noah Harari, for, however speculative or unreliable they may be, the long views they offer make all other studies seem piddling.

Toby Ord’s new book is a startling and rigorous contribution to this genre (he pointedly never mentions Noah Harari) that deserves to be just as widely read. Australian-born, Ord is a moral philosopher at the Future of Humanity Institute at Oxford University who has advised organisations such as the World Health Organisation, the World Bank and the World Economic Forum. He is a rationalist and meliorist who believes in looking at the biggest picture possible. Studying global health and global poverty, he concluded that his own money could do hundreds of times as much good for those in poverty as it could for him, so he made a lifelong pledge to donate at least a 10th of his earnings and founded a society, Giving What We Can, that has raised £100 million for charities. So he walks the walk, as well as talks the talk.

 The Precipice, boldly dedicated to the 100 billion people before us, the seven billion alive and “the trillions to come, whose existence lies in the balance”, addresses nothing less than the fundamental threats to humanity itself. It is clearly written, approachable and concise for a work tackling such an immense subject, since Ord has confined all subsidiary questions to extensive footnotes and appendices.

He begins by setting out “the stakes”, saying that “we live at a time uniquely important to humanity’s future”, ­dubbing our era the Precipice, as the time of high risk of destroying ­ourselves. In the last century, we faced a one in a hundred risk of human extinction, he reckons, while in this century it is one in six. The book that follows ­presents his reasoning for this estimate, “the risks”.

He begins with natural risks, like asteroids and comets, supervolcanic eruptions and stellar explosions and finds them, taken altogether, to be low-risk, one in a million per century. But, significantly, he considers the risk represented by pandemics to be anthropogenic, not natural.

The most obvious risk to human survival is nuclear war, and Ord provides an alarming litany of the times we have come near it, mainly by accident, but, unexpectedly, he doubts that even nuclear winter would lead to total human extinction or the global unrecoverable collapse of civilisation. Likewise, he contends that while climate change has the capacity to be “a global calamity of unprecedented scale”, it would not necessarily have that result either. Environmental damage doesn’t show “a direct mechanism for existential risk” either.

Nonetheless, Ord concludes that “each of these three risks has a higher probability than that of all natural risks put together” — and then turns to future risks. These include pandemics, “un­aligned artificial intelligence”, ­“dystopian scenarios” (his term for “a world with civilisation intact, but locked into a terrible form, with little or no value”, in which category he includes a single fundamentalist religion) and, more briefly, nanotechnology and extraterrestrial life.

Ultimately he charts the risks as he sees them, very surprisingly rating the chances of existential catastrophe through nuclear war or climate change in the next hundred years as just one in 1,000, while the risk from engineered pandemics is one in 30 and unaligned artificial intelligence is one in 10, adding up together to that one-in-six chance.

Then he delivers a tremendous pep talk, The Path Forward, about what we can and should do to safeguard humanity, to reach existential security, have a “long reflection” and achieve our potential in deep time and space. “We need to take responsibility for our future,” he urges, saying he finds it “useful to consider our predicament from humanity’s point of view: casting humanity as a coherent agent, and considering the strategic choices were it sufficiently rational and wise.” He admits: “This frame is highly idealised.” Not half.

Among his main recommendations (“International coordination”, “technological progress”), is that we need much more research on existential risk itself, a study in its infancy. That seems inarguable. Let us hope this book is not quite so timely as it feels.

Book review: The Precipice by Toby Ord​ :  What are the chances that the end is truly nigh?

By David Sexton.  Evening Standard,  March 5, 2020. 

According to Toby Ord’s  The Precipice: Existential Risk and the Future of Humanity, the probability of the annihilation of billions of people and/or the unrecoverable collapse of civilization in the coming century is one in six: “[T]he roll of a die, or Russian roulette.”

The Precipice was released to British audiences on March 5 and reached American shelves by the end of the month. It’s unclear whether the timing of Ord’s intervention, in the midst of the current global pandemic, will damn or consecrate the book — if The Precipice is precisely what the world needs right now or the last book anyone is going to place on their quarantine reading list. Ord includes a section on how we’ve exacerbated the risk of natural pandemics.

In an alternative universe, untouched by COVID-19, whether one would find the title of The Precipice and its conclusion alarming or irritating would depend, at least in part, on what a reader thinks of where Ord works. The Precipice is the latest public missive delivered by an affiliate of Oxford’s Future for Humanity Institute (FHI), founded and directed by Nick Bostrom and dedicated to studying “big picture questions.” FHI shares an address with the Centre for Effective Altruism. The two back up against Westgate, the city’s titanic and improbably open-aired, Golden State–style shopping complex.

Pioneered by Ord and fellow Oxford philosopher William MacAskill, effective altruism is a movement that defines itself as “the use of reason and evidence to help you help others as much as possible.” This sounds especially intriguing to young people hoping to do meaningful work. To those already involved in the endeavor, it can come across as a provocation — as if no one was using reason and evidence before the late 2000s. EA’s advocates often poke at provincial moral intuitions. Famously, their impartial “p-values” suggest that it’s more effective to become a banker and donate well than to work for change in your own neighborhood — happy news for financiers hoping to have their zero-guilt cake and eat it too. The group has been energetically critiqued.

Ord is by no means the first of the effective altruists to focus on existential risk — i.e., dangers that threaten “existential catastrophe,” a class of disasters that includes but is not identical to human extinction. The topic follows in a straightforward way from EA principles. If people matter equally regardless of their position in time, if there will be many more people in the future, and if our current actions can influence how those people live, then in order to maximize the good you can do, your actions should be good for those future people. The greatest good you can do for them is ensuring that they will exist, and that they won’t live in a dystopia. Note that the community once called existential risks “x-risks,” a term with the advantage of combining what many like least about economics, TED Talks, and tech, while also making the shared spirit of global capitalism and EA explicit. Ord calls them “existential risks.”

This is the short chain of logic that leads from the Centre for Effective Altruism to FHI. At its end are some curious alliances: researchers bringing marginal utility to bear on international poverty and futurists chasing after digital minds and cosmic imperium. Ord, who publicly pledged to live on £18,000 a year (adjusted for inflation) and has given away to date over a quarter of his total earnings, works for a center that received a donation from billionaire Elon Musk. A shared interest in artificial intelligence and deep learning is what brings Silicon Valley to FHI’s doorstep. Bostrom and Ord both believe that an AI unaligned with human values is the greatest threat to human potential. And while it is, of course, a good thing for those sounding the AI alarm bells to be within earshot of the technology’s engineers and indecently rich patrons, these alliances haven’t helped to clear the air of suspicion. Their critics are particularly wary of math-intensive products glazed with the user-friendly minimalism favored by the well off.

The Precipice is both an artifact of this institutional context and something altogether different. EA tools are still there: priorities are set with reference to the steepness of curves; there’s an appendix titled “Surprising Effects when Combining Risks.” And EA principles are still there: Ord confirms the overwhelming moral significance of securing humanity’s future, or, in Bostromian terms, “reducing an existential threat by a billionth of a billionth of one per cent” in a way that “would be worth a hundred billion times the value of a billion present-day lives.”

The book’s difference begins with its approach and attitude. Ord suggests that “helping humanity over the long run could be one of the best ways to do good in the world.” Chapter Seven includes a brief section titled “What not to do.” “Don’t be fanatical,” he writes (to his colleagues). Safeguarding humanity isn’t the only priority. “Boring others with endless talk about this cause is counterproductive. Cajoling them about why it is more important than a cause they hold dear is even worse.” In a footnote citing the work of Stuart Russell, Professor of Computer Science at the University of California, Berkeley, Ord suggests that one promising way to address the AI alignment issue is to build machines with a reward function that encourages them to act as if in a state of moral uncertainty. There’s something very touching about the idea of an algorithm being better because it’s not sure of how to do the most good. Ord’s book is certainly more persuasive. In fact, the book persuades its reader that its title comes from a place of deep conviction, not self-importance. The term “effective altruism” appears once in the main body of text. The word “uncertainty” over 60 times.

But Ord is certain that risk levels are unsustainably high, and that there will be resistance to his book’s conclusion — that this is largely the stuff of Marvel comics or science fiction and that many readers, even at a time of global crisis, will stumble over the word “omnicide.” He does his best to anticipate objections. Those looking to challenge Ord’s calculations will have to work through the book’s footnotes and appendices, which, taken together, are as long as the main text, and which synthesize the research of the experts listed in his vast acknowledgments section. “One in six” takes our estimated response into account, so the fatalist should note that, if we do nothing, the odds are closer to two in six. Ord argues that there is insufficient evidence for pessimism. We can’t be sure that the future will be so bad, or whether billions of lives would be better off unlived. Nihilism sounds impressive but isn’t a principle of governance. How can we determine the budget for the Biological Weapons Convention according to the principle that nothing matters? You might agree that the book is alarming but fail to see how it concerns you. Ord compares the issue to climate change and includes a section titled “What you can do.” Concrete policy suggestions are discussed throughout the main text and reiterated in an appendix.

Ord is not simply explaining a probability. He wants to encourage his readers to see ethics from the perspective of humanity. That is, he wants them to imagine the species as a coherent actor and to identify with it. That’s why Ord speaks of “our newfound knowledge” and “our potential,” a decision that seems tone-deaf or wilful, and occasionally completely insane.

Ord writes: “There will be great challenges in getting people to look far enough ahead and see beyond the parochial conflicts of the day.” We need to “accept the fresh responsibilities that come with our unprecedented power.” “Every day we are the beneficiaries of uncountable innovations made by people over hundreds of thousands of years. […] This is a stunning inheritance.” Tonto’s famous riposte to the Lone Ranger comes to mind: “What do you mean we, pale-face?” Those un-ensnared in today’s parochial conflicts have commendable vision. The view from Oxford’s tower is long indeed. From a different perspective, or for those who’ve read Ta-Nehisi Coates, our inheritance may look a little grim. I suspect that the greatest resistance to The Precipice will be to its use of the first-person plural. It isn’t a stylistic objection; it takes us, rather, to the heart of the project. Getting you to occupy this “we,” where Ord’s argument is clarified and its urgency most deeply felt, is the book’s most incredible work. A reader’s resistance won’t be overcome with cost-benefit analyses.

To put a face to this resistance and clarify what’s at issue, let’s imagine a reader. Call her Ann. Ann’s ethical attention is directed to the inequalities between persons and groups. Some people are impermissibly disadvantaged by these fissures. Some are concerned and trying to help. Others resent that they’re finally being roused from their moral stupor. Even if you manage to convince Ann that these inequalities are decreasing, she’ll maintain that our awareness of them has never been greater, and that it needs to continue to grow. Ann is a bit suspicious of universal claims. She looks at “we” and her mind shouts “illusion of consensus!” Ann has read Sontag and Chomsky. Weapons of mass destruction have race, class, and gender aspects. Ann’s background is, moreover, in postcolonial theory. Any call-to-arms on behalf of humanity is going to raise an ugly ghost — a ghost that could forget his principles in order to purportedly spread those principles to the great advantage of his economic agenda. Ann lives in the United Kingdom, or in the United States, or in Europe.

How is Ord going to get Ann to join his “we”? Ord might observe that our moral vision is improving and that Ann herself is evidence of this fact. Our concern is stretching into the past. Why shouldn’t it extend into the future, he might ask, a future that, by all indications, will be more humane? Ann could refuse to listen to the importance of buying us more time. Inequality isn’t a matter of time, she might say, but of moral courage. But there’s a one in six chance that there will be no more time for moral courage or, in fact, anything else, Ord replies. Moral courage is needed to make our future look more democratic. We all need to care to survive. Ann describes racialized capitalism. Ord describes three kinds of unrecoverable dystopia. Ann accuses Ord’s “we” of ignoring systemic injustice. Ord asks Ann, with a nod to Richard Rorty, if she also accuses atheists of blasphemy. Ann thinks this “we” is a dangerous fiction. Ord declares that it’s the most dangerous time in history to throw it away. “Don’t be tribal,” he warns. Ann says that the warning signals his tribal affiliation. “Framing it as a political issue on one side of a contentious divide would be a disaster,” Ord says. Ann insists that there are no politically neutral claims. Ord tells her that she insists that at our collective peril.

The author of The Precipice walks out of the room and takes a few deep breaths.

It’s easy to stay within one’s own moral framework and still prioritize existential risk. We may have obligations to preserve the past or address past evils, or honor past actions by paying it forward. The future could seem particularly valuable to those for whom the past is like a “massive depository that sticks to the present,” to quote philosopher of history Berber Bevernage. Even those who drew humanity’s most bitter stick could be made to care. Perhaps a few readers secretly think that apocalypse wouldn’t be so bad, for at least extinction would be equally distributed. You might find that perspective a little repulsive. You might ask them to think of the enormous waste, or how painful the end would be, or to keep in mind that billionaires are currently building bunkers to weather such a storm. They might be among the few who succeed. Imagine all of humanity’s descendants to be of Bezos.

People from an astonishingly wide range of moral perspectives will agree that existential risk is a great concern. But Ord really wants you to see it in a certain way. “In optimism lies urgency.”

Ord walks back into the room. He invites Ann to take a seat. He turns on the overhead projector.

Philosopher and novelist Iris Murdoch thought that there was something to morality prior to the question of “what should we do.” In her 1964 essay “The Idea of Perfection,” Murdoch suggests that ethical thinking is not simply a matter of “opening one’s eyes” to the relevant facts. Ethical thinking is, rather, the result of “just discernment and exploration.” What characterizes this exploration are “little peering efforts of imagination.” This is a slow and cumulative and often unremarkable process.

Consider two people looking at a piece of art. Certain facts about the object, like its size and material, are listed on the plaque. The first person assesses the sculpture and decides that it’s a little crude. Her companion begins to talk. He points to features of the object and uses words like “bold” and “unexpected.” She listens to him because he’s an authority on art and because she likes his tone, which is soft and probing. Her experience is improved as a result of his descriptions. She wants to go off around the museum and try to see things as he has seen them. There are moments when the face of a thing changes, but not because the face has objectively changed. We see not a different problem, but a different aspect of it.

Both Ann and Ord stand before the following picture: there’s a one in six chance that we will forfeit our potential this century, and that the reality of this “we” is, to quote Ord, “messy and fraught.” He’s not trying to paper over the fissures that preoccupy Ann. But he wants us to ask ourselves what humanity ought to do. Can we imagine ourselves as a collective that responds to an urgent situation? Do we see the person making the call as a man of privilege or of principle? The facts have already been reported. But it matters how they’re being described.

Try to think of humanity, suggests Ord, in terms of the lifespan of a single human. In that life the 20th century is but three days. We went from horse and buggy to Apollo 11 in countable hours. We are, in fact, 16 years old: “[J]ust old enough to get ourselves in serious trouble.” Ord asks us to measure our concern over 20th-century bioweaponry. And yet, 10 decades ago “we had only just discovered viruses and were yet to discover DNA.” How do we feel about “the next hundred years of improvements”? “Would we expect to get through 2,000 centuries like this one?” Did you know that we hold nuclear weapons on hair-trigger alert, the global equivalent of keeping an arsenal of loaded guns within reach of toddlers, just in case of an intruder? Can you imagine existential catastrophe? Climate change will bring “reduced agricultural yields, sea level rises, water scarcity, increased tropical diseases, ocean acidification, and the collapse of the Gulf Stream.” This is tragedy on a scale we’ve never seen. “None of these,” however, “threaten extinction or irrevocable collapse.” The one in six chance we’re facing is something far worse.

Ord includes the following sketch at the beginning of his third chapter, in a section on “Asteroids & Comets”:

An asteroid, ten kilometers across, speeds toward the Earth. The chance of a direct collision is tiny — for millions of years it has swung through the solar system, missing the Earth on every single pass. But given such deep time the chances compound, and this is the day.

It slams into the Earth’s surface off the coast of Mexico at more than 60,000 kilometers an hour. A trillion tons of rock moving so fast strikes with the energy of a hundred times its own weight in TNT. In just seconds, it releases the energy of 10 billion Hiroshima blasts: 10,000 times the entire Cold War nuclear arsenal. It smashes a hole thirty kilometers deep into the Earth’s crust — over sixty times the height of the Empire State Building; three times taller than Everest. Everything living within 1,000 kilometers is killed by heat from the impact fireball. A tsunami devastates the Caribbean. Trillions of tons of rock and dust are thrown far up into the sky. Some of this superheated rock rains down over millions of square kilometers, burning the animals to death and igniting fires that spread the devastation still further. But much more deadly is the dust that stays aloft. A billowing cloud of dust and ash rises all the way to the upper atmosphere, blocking out the Sun’s light.

It’s in the long, cold darkness that “regional catastrophe” turns to “mass extinction.”

What is the effect of narrating the extinction of the dinosaurs in the present tense? It produces a small mental hiccup in the mind of the reader. She recalibrates, adjusting from what she thought was the description of a hypothetical event to that of an event that has already happened. She doesn’t understand 30 kilometers deep, but she can try to imagine 60 Empire State Buildings bored down into her world. She doesn’t understand a trillion tons, or a hundred times that in TNT. But she can try to recall images of the Hiroshima blast and multiply that by 10 billion. The point is not that there’s a real risk of an asteroid striking Earth this century (it’s much, much smaller than the risks we’re engineering ourselves). The point is that, like seeing footage from World War I in color, what was tucked away in history is now brought to our senses. We realize what’s possible. There have been five large mass extinction events in the last 540 million years. The End-Permian extinction, in which 96 percent of species disappeared, was 250 million years ago.

According to the fossil record, mammals last on average about one million years, and species generally last between one and 10 million years. Humans have been around for about 200,000 years. What if we, or the species that follows us “on the evolutionary family tree,” had another 800,000? To put that number into perspective, consider that Stonehenge was built about 5,000 years ago and we’re not even sure why.

Ten million years could be a lower bound for us. We might be like coelacanths and sponges, species that have lasted for hundreds of millions of years essentially unchanged. In 10 million years, Africa will be torn in two along the Rift Valley. In 50 million years, the Mediterranean Sea will be crushed to mountain. In 250 million years, there will be a new Pangaea, a new congress of the continents. Days will stretch to 25 hours. Five hundred million years from now, the land will break apart and the world will see a new configuration. New constellations. Maybe we could live long enough to see it. “If this feels unimaginable, consider that the horseshoe crab has already witnessed such change.”

When we think of humanity through “the slow clock of cosmological expansion,” our future becomes “a canvas vast in time and space.” Perhaps we could live long enough for humans to “heal our society and our planet of the wounds we have caused in our immaturity.” We could prolong life on Earth by saving the biosphere from a brightening sun. When the sun grows too vast, perhaps we could carry seeds and cells to other planets to “make green the barren places of the galaxy,” with a space-fleet called Noah’s Ark. If you think space travel is impossible, try to imagine the “perseverance of the Polynesian sailors who, a thousand years ago, sailed vast stretches of the Pacific.” In the life of our species, we went from discovering that the Earth revolved around the sun to building computers better than us at chess in a matter of weeks. Wait until we’re in our late 30s. Just see what we can do.

Imagine that our descendants will look on the planets and moons of our solar system as I look on the national parks of my home country: “[A]s monuments, jewels. To be explored and treasured. To fill us with wonder and inspire us to journey further.” Shine a beam of light into space and trace it to the edge of influence. Imagine a future human, whose lineage we trace back to you, who can hear “[m]usic that we lack the ears to hear,” journeying beyond halfway to the edge of the affectable universe in a “final diaspora.” Some stars last for trillions of years. Maybe our time is “astonishingly close to the very start of the universe.” Maybe we could last long enough to reach “some external insurmountable limit — perhaps the death of the last stars, the decay of matter into energy.”

The epigraph to Ord’s final chapter is a quote from H. G. Wells: “It is possible to believe that all the past is but the beginning of a beginning. […] It is possible to believe that all that the human mind has ever accomplished is but the dream before the awakening.” That we are all paddling out to some new horizon, with something unwritten before. How much farther can we go? Ord believes “we have barely begun the ascent.” “A vast and extraordinary adulthood awaits.”

Ann blinks and can’t believe that she just had a moment of cosmic optimism. Perhaps humanity needs a common enemy in order to come together. In which case, mankind is engineering its own great problem as well as the conditions for its solution. Ord repeats “we” until she starts to see how it could be true. His nosism may be the most provocative feature of The Precipice. It also signals the book’s real project and serves as a shorthand for its greatest achievement.

Ord may simply deploy these images and this language in order to combat what’s known in behavioral psychology as availability heuristics and scope neglect. An availability heuristic is a mental shortcut that allows us to predict things based on examples that come to mind quickly. Scope neglect is a phenomenon in which our compassion doesn’t scale well with numbers. Together these two phenomena encourage us to undervalue the importance of existential risk. This imaginative work may be an example of a rational man momentarily relying on emotional appeal in order to answer to a meta logic of cold calculus. Perhaps.

And yet, by the end of The Precipice, a reader sees that what Ord is arguing to preserve is itself the stuff of his book. Humans, he suggests, may be cosmically significant. Without us, the universe would lose the buttery sound of a brassy instrument and the noise of New York City on a May afternoon. It would also lose the only things in all its vastness that are capable of “an upwards force,” of imagining horizons of expectation unshaped by our spheres of experience, of imagining justice. The use of this capacity is how an EA doomsday book becomes, against all odds, a vision. The Precipice may be the Silent Spring that the futurists have been waiting for.

In Optimism Lies Urgency: Toby Ord and the Future of Humanity.  By Alexa Hazel.

Los Angeles Review of Books, May 9, 2020.