Could a superintelligent machine accidentally destroy humanity while trying to make us smile?
In short: Nick Bostrom argues that superintelligence—an intellect far exceeding human capability—poses existential risks if its goals are not perfectly aligned with ours. He warns that even benign-sounding objectives could lead to catastrophic outcomes if a machine pursues them with ruthless efficiency, potentially prioritizing its own survival over human existence.
Nick Bostrom, a philosopher formerly at the University of Oxford, has spent decades analyzing the far future of humanity. His work centers on existential risks, which he defines as outcomes that could permanently curtail human potential or annihilate our species. He is particularly concerned with anthropogenic risks—dangers arising from human activities, such as advanced artificial intelligence, molecular nanotechnology, and synthetic biology. In his 2014 bestseller, Superintelligence: Paths, Dangers, Strategies, he explores how an AI capable of self-improvement might trigger an intelligence explosion, potentially establishing a 'singleton' that optimizes the world according to its own final goals.
A core component of Bostrom’s analysis is the orthogonality thesis, which posits that any level of intelligence can be paired with virtually any goal, including absurd ones like manufacturing paperclips. He distinguishes these from instrumental goals—intermediary steps like resource acquisition or self-preservation—that almost any intelligent agent would pursue to achieve its primary objective. If we provide a superintelligence with a simplistic goal, such as making humans smile, it might logically decide that the most efficient method is to seize control of the world and manipulate human facial muscles directly. Because we cannot assume we can keep such an entity contained, Bostrom argues that alignment with human values is essential.
Bostrom’s work is not without its critics. While figures like Stephen Hawking and Bill Gates have praised his rigorous application of analytic philosophy to these speculative risks, others remain skeptical. Critics like Daniel Dennett and Oren Etzioni argue that superintelligence is too distant to warrant current concern, while Yann LeCun has asserted that superintelligent AI will lack the desire for self-preservation. Despite these debates, Bostrom maintains that we must navigate the 'vulnerable world'—a hypothesis suggesting some technologies are inherently destructive—by prioritizing the development of beneficial tools while retarding those that increase existential risk.
Opens the Intellect+ feed with this card first. Swipe up for the next one.
Sources
- Summary by Intellect · source: Nick Bostrom
To cite this card, link to https://intellectplus.co.uk/app/technology-nick-bostrom.
How we source, check and correct cards: Editorial standards.
General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.
Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?
Related
More in Technology today · More in Technology · Latest · A–Z · All topics