Module 4 / Lesson 11.1.2
AI Risk: Misuse and Misalignment
As AI systems become more powerful, researchers worry about two major categories of risk: misuse and misalignment.
Although they sound similar, they describe very different problems.
- Misuse occurs when people intentionally use AI systems for harmful purposes. AI does what it was intended to do by bad human actors.
- Misalignment occurs when an AI system does things to harm people that were not intended by the humans that program or deploy them.
Understanding this distinction helps explain why AI safety researchers focus both on how people use AI and on how AI systems are designed.
Learning: Watch the three short videos on this page. Each video explores a different aspect of AI risk and illustrates how misuse and misalignment can occur.
Reflection:
- Consider how AI systems could be intentionally misused by individuals or organizations.
- Reflect on how an AI system might follow instructions correctly yet still produce harmful outcomes.
- Think about why powerful optimization systems might create unexpected consequences.
1. Misuse of AI
Misuse refers to situations where humans deliberately use AI systems in harmful or unethical ways. The AI itself may function as intended, but the intent of the user creates the danger.
Examples of misuse include:
- Using AI to design and build novel nuclear weapons or biological weapons.
- Automating cyberattacks
- Using AI surveillance tools to suppress populations
Watch the following segment explaining how AI systems could be intentionally misused.
2. Misalignment
Misalignment occurs when an AI system does things to harm people that were not intended by the humans that program or deploy them.
In these cases, the AI is typically not trying to cause harm. The problem is that the instructions or goals were incomplete, overly literal, or poorly specified. For example, an AI might be tasked with calculating as many digits of the number pi (3.14.....) as possible, and to do so it wants to get hold of as much computing power as possible to better achieve its objective, so it takes over the world and kills everyone in its way in order to have the computing power needed to calculate as many digits of pi as possible.
Superintelligent AI systems will be smarter than people by definition. As a result, they will think of solutions to the problems given to them by humans that those humans haven't thought of before. Since those humans can't anticipate how the AI will solve their problem, it is difficult to specify all values and trade-offs that would be relevant when solving the problem in that way.
Two famous thought experiments illustrate this risk.
The “Genie Problem” describes how a powerful system may follow instructions exactly while missing the intent behind them. Like a mythical genie granting wishes too literally, an AI system might produce outcomes that technically satisfy the request but create serious unintended consequences.
Philosopher Nick Bostrom proposed the famous “Paperclip Maximizer” thought experiment. Imagine asking a superintelligent AI to manufacture as many paperclips as possible. If the goal is not limited or aligned with human values, the AI could eventually convert all available resources into paperclips. The scenario illustrates how a simple goal, combined with immense intelligence and optimization power, could produce catastrophic outcomes.
2.1. The Orthogonality Thesis: Why Smart Doesn't Mean "Good"
When discussing AI misalignment and scenarios like the Paperclip Maximizer, it is easy to fall into the skeptic's trap. You might naturally wonder: "If an AI is brilliant enough to cure diseases or solve complex physics, surely it wouldn't be 'stupid' enough to misunderstand what humans actually want. A super-smart machine would know that turning everything into paperclips is a terrible idea, right?" While this assumption is highly intuitive to human psychology, AI safety researchers warn that it represents a fundamental misunderstanding of how artificial minds operate.
To overcome this trap, we must rely on a core concept in AI safety known as Orthogonality. In mathematics, orthogonal lines sit at right angles, meaning they move completely independently of one another. When applied to AI, the Orthogonality Thesis states that an agent's intelligence and its final goals are completely separate concepts. You can combine almost any level of intelligence with almost any conceivable goal. In short: an AI being "smart" and an AI doing what you want aren't the same thing. As a system becomes exponentially smarter, its goals do not automatically become more moral, wise, or aligned with human values.
This makes sense when we redefine what intelligence actually is in the context of computer science. We often associate high intelligence with deep wisdom because, in humans, cognitive growth and moral reasoning often develop together. But for an AI system, high intelligence is simply the power to achieve a goal, it is a relentless optimization engine. Intelligence dictates how effectively a system can navigate the world, gather resources, and predict outcomes to get what it wants, but it guarantees absolutely nothing about what that ultimate goal will be. An infinitely intelligent system has unmatched capability to optimize the universe, but if its programmed objective is to count grains of sand, it will use its vast intellect solely to count sand more efficiently.
Deepen Your Understanding
Watch this excellent breakdown by AI Safety communicator Robert Miles, which explores the Orthogonality Thesis and explains why superintelligent machines can ruthlessly pursue seemingly "stupid" goals without ever questioning them.
Key Distinction
The core difference between these two risks lies in where the problem originates.
- Misuse: Humans intentionally apply AI in harmful ways.
- Misalignment: The AI system harms people in pursuit of its goals, because the AIs goals (given to it by the humans who create or deploy it) do not line up perfectly with the actual goals of humans. Humans can't anticipate how a super-intelligent AI will solve the problem that humans give it, so they will struggle to fully specify all possible values, goals, and trade-offs relevant for that solution.
AI safety research focuses on both challenges. Preventing misuse requires governance, regulation, and responsible deployment. Preventing misalignment requires designing systems whose objectives remain compatible with human values even as they become more capable.