Module 2 / Lesson 5.3

Public Goods and Racing Problems in Existential Risks

5 min read

Why is it so hard for humanity to cooperate to prevent existential risks, even though everyone benefits? In Lesson 5.1, you learned about public goods, and in particular about why we might get too few of them in settings with free-riding where individuals do what is best for them. In Lesson 5.2, you learned about strategic interactions such as the Prisoner's Dilemma in game theory, which was another example of why we might get a socially bad outcome in a strategic setting where individuals do what is best for them. In this lesson, we explore why society might not do enough to prevent existential risk, for reasons including the free-riding you learned about in Lesson 6.1 and the Prisoner's Dilemma you learned about in Lesson 5.2, plus a few other reasons mentioned in by Toby Ord in The Precipice.

📚 Learning Tasks

Look at Lesson 5.3.1 below on Public Goods as Existential Risks to learn why we might think of existential risk prevention as a public good that we might be inclined to do too little of
Look at Lesson 5.3.2 below on Racing Problems in Existential risk to learn why individual companies or countries might want to race to be the first to produce AI or medical breakthroughs, even if working in these areas is risky for humanity and we would collectively better off if no one worked in these risky areas.
Read Ord pp. 57-64 (which is Lesson 5.3.3) to see why he thinks there is a tendency to put in too little effort on the prevention of existential risk, including the two reasons from Lessons 5.3.1 and 5.3.2.

🤔 Thinking Tasks

As you are learning about the challenges that make it hard to put sufficient resources into preventing existential risk, think about the barriers that make it hard to create enough lighthouses or that make it hard to cooperate in the Prisoner's Dilemma.
As you are reading Ord, think about the key reasons he thinks why it may be hard to put sufficient resources into preventing existential risks.

📖 Lesson 5.3.1 Existential Risk as a Public Good

Ord considers that the prevention of existential risk is the ultimate public good, everyone benefits, whether or not they contribute.
Key Point: Existential risk reduction is non-excludable and non-rivalrous. Preventing existential risk benefits everyone (non-excludable), and one person enjoying protection from existential risk doesn't prevent others from enjoying it.
Preventing existential risks is in everyone's interest, but no individual, group, or country has much of an incentive to spend resources to reduce existential risk since they bear the full cost while the benefits accrue to everyone (they receive only a tiny fraction of the full benefit). Everyone is inclined to free-ride on the existential risk prevention activities of others, since people benefit even when they don't pay, so too little prevention of existential risk happens.
Nick Bostrom (https://existential-risk.com/concept.pdf, Chapter 4) notes that researchers publish vastly more research on dung beetles (and snowboarding and Star Trek) than they do about existential risk, even though existential risk is likely a more important topic.

Table: Existential Risk as a Public Good

	Ordinary Public Good	Existential Risk Reduction as Public Good
Example	Build a lighthouse	Investing in a telescope to look for asteroids heading for earth, so you can redirect them if needed
Excludable?	No (all ships can use the lighthouse)	No (all humans benefit from this information, whether they pay for a telescope or not)
Rival?	No	No (one group using the telescope does not prevent others from benefiting from this knowledge)
Potential Payer	Owner of seaside land	Canadian government
Who pays?	Owner of seaside land	Canadian government
Who benefits?	All ships	All people
Outcome?	Too few lighthouses	Too little activity looking for asteroids heading for Earth

🌍 Lesson 5.3.2 Racing Challenges on Existential Risk Reduction

Many companies and countries are racing to be the first to develop super-intelligent AI; being first would lead them to have a great deal of profit and influence.
However, we will learn in Lesson 11 that developing super-intelligent AI may be dangerous.
Any individual company or country benefits from pushing ahead with AI development, but the dangers of doing so impose costs on everyone else.
As a result, we might see many companies or countries racing ahead to develop AI, even if it were better off if no one did.
If no one else is developing AI, you want to work to develop AI to benefit yourself (and you ignore the possible harm to the rest of humanity if your efforts go awry). If others are developing AI, you want to develop AI so you are not left behind (and you ignore the possible harm to the rest of humanity if your efforts go awry). Therefore, it may be individually optimal for any given country or company to pursue super-intelligent AI even if everyone would be better off if no one pursued super-intelligent AI.
This setting looks a lot like the Prisoner's Dilemma from Lesson 5.2, where one prisoner defecting/testifying against/betraying their colleague is analogous to one company working to develop super-intelligent AI. Both prisoners would be better off if neither testifies against the other, but both choose to testify against the other since they benefit from testifying at the expense of the other prisoner. Similarly, perhaps all countries would be better off not working to develop super-intelligent AI, but many choose to work to develop super-intelligent AI since they benefit from developing AI at the expense of the harm that existential risk from AI poses to all of humanity.

Table: Racing to Develop Technology With Existential Risk as a Prisoner's Dilemma

	Ordinary Prisoner's Dilemma	Existentially Risky Technology Development Risk as a Prisoner's Dilemma
Agents	Two prisoners	Countries working to develop super-intelligent AI
Selfish Action	Defect/betray/testify against another prisoner	Work to develop super-intelligent AI despite its existential risks
Cooperative Action	Do not betray/testify against other prisoners	Halt work to develop super-intelligent AI
Benefit of selfish action	Shorter prison term for self	Wealth or power that comes from being first to develop AI or not being left behind in AI development
Harm of selfish action	Longer prison term for another prisoner	Developing super-intelligent AI could kill all of humanity, almost all of whom do not live in your country.
Socially Preferred Outcome	Do not betray/testify against other prisoners	Halt work to develop super-intelligent AI
Individually Optimal Outcome	Betray/testify against another prisoner	Developing super-intelligent AI that could kill all of humanity

Other examples of racing and existential risk:

Nuclear weapons: The “security dilemma”, everyone fears being second to develop nuclear weapons, or to not have nuclear weapons when others do, so everyone races to develop nuclear weapons, even though the threat of nuclear war means it might be safer if none had nuclear weapons.

💡 Lesson 5.3.3 Ord on why efforts to prevent existential risk are too few

Read Ord pp. 57-64 on why he thinks there are too few efforts to prevent existential risk.

Your progress stays in this browser.