Module 4 / Lesson 11.1.3
What Experts Think
This section looks at what experts think about the potential for artificial intelligence to pose an existential risk. Specifically, it covers two different ways that expert views are captured:
- Large-scale surveys of experts, which give us a broad picture of the community’s expectations and concerns.
- The individual estimates of specific experts, often summarized in the form of P(doom) scores.
Recall from Lesson 1.1, we defined risk based on frequenccy, severity and scope, the same way P(doom) refers to the probability that AI development could lead to human extinction or another existential catastrophe. In this section, we focus not on the definition but on how experts actually use it in their reasoning.
Learning: Read carefully Sections 1 and 2 below in this page on expert assessment of AI risk and the views of specific famous people on their p(doom). Be sure to click on the “Thousands of Authors on AI” link referenced below and to look at what they say about the risks of AI to understand the distribution of expert assessments.
Reflection: As you read, think about
- What could be the two specific reasons or mechanisms (e.g., unintended consequences, misuse) an optimist might still give a significant probability to a catastrophic outcome?
1. Expert Assessments of AI Risk (Survey Data)
Surveys are one of the most direct ways of gathering expert opinion. For example, Katja Grace and colleagues published the paper “Thousands of Authors on AI”. This large-scale study gathered input from 2,776 AI researchers, asking not only about timelines for AI capabilities but also about the likelihood of catastrophic or existential outcomes.
Key takeaways from the survey include:
- The aggregate forecast for High-Level Machine Intelligence (HLMI) reaching 50% probability is now estimated by 2047, down 13 years from the 2022 survey. For Full Automation of Labor (FAOL), the median forecast dropped by 48 years since 2022.
- Median respondents put a 5% or more chance on advanced AI leading to human extinction or similar extremely bad outcomes.
- A third to a half of participants gave a 10% or more chance to these extremely bad outcomes across four related questions.
- Researchers are now expected to be professionally fully automatable a quarter of a century earlier than in 2022. The forecast for AI-generated NYT bestselling fiction dropped by more than half, to around 2030.
-
For every one of eleven specific AI-related scenarios (including spread of false information, authoritarian control, and worsened inequality), at least a third of participants found it deserving of substantial or extreme concern.
-
While 68% of participants thought HLMI was more likely to lead to good outcomes than bad, nearly half of these "net optimists" still put at least a 5% chance on extremely bad outcomes.
-
70% of participants would like to see research aimed at minimizing the risks of AI systems prioritized more highly, with a third requesting "much more" prioritization.
2. Famous Voices and P(doom)
Beyond surveys, much of the debate about AI risk comes from the public estimates of individual experts. These are often expressed as P(doom percentages) , shorthand for how likely they think it is that AI could cause an existential catastrophe.
There is no consensus: some well-known figures assign a very low probability, while others suggest probabilities above 50%. Comparing these estimates shows the depth of disagreement and the uncertainty in forecasting such unprecedented risks.
To see this diversity, review the Wikipedia summary of P(doom), which compiles estimates from leading AI voices.
One of the earliest and most vocal proponents of high P(doom) estimates is Eliezer Yudkowsky, a foundational thinker in AI safety. In his TED Talk, he outlines the core arguments for why we should be deeply worried about artificial general intelligence (AGI). He suggests that unless we fundamentally change our approach to alignment, the default outcome of creating an unaligned, superintelligent AI will be human extinction.
Watch Eliezer Yudkowsky's TED Talk below. Pay close attention to his argument about why relying on "trial and error" for AI safety won't work when dealing with superintelligence.