Can we prevent superintelligent AI from causing human extinction before it is too late?

Can we prevent superintelligent AI from causing human extinction before it is too late?
Image: Intellect (generated)

In short: Nate Soares, president of the Machine Intelligence Research Institute, argues that current AI development is moving too fast for safety. Once a leading technical researcher on AI alignment, he now focuses on public awareness, warning that creating systems far more capable than humans could lead to catastrophic global outcomes.

Nate Soares is a central figure in the discourse surrounding existential risk from artificial intelligence. After a career in software engineering at Microsoft and Google, he transitioned to the Machine Intelligence Research Institute (MIRI) in 2014. His early work focused on the technical challenge of AI alignment—the problem of ensuring that increasingly capable systems act according to human intent. In 2015, he became MIRI's executive director, where he led research agendas exploring the risks of rapid self-improvement in machines, or intelligence explosions.

The scope of Soares's work shifted significantly in 2023. While MIRI continues to support technical research, the organization pivoted toward warning the public and policymakers about the dangers of future AI developments. This shift culminated in the 2025 book If Anyone Builds It, Everyone Dies, co-authored with Eliezer Yudkowsky. The authors contend that building superintelligence using current techniques will likely result in human extinction. They argue that alignment research is still in its infancy and that international regulation is necessary to prevent a dangerous race among developers.

The reception to this alarmist perspective has been varied. While some outlets like The Guardian highlighted the clarity of the book's argument, others, such as The Washington Post, criticized it as a polemic lacking concrete, actionable instructions. Soares himself has acknowledged this change in focus, telling The Atlantic in 2025 that he has prioritized public awareness over technical research. He maintains that the pace of AI advancement is currently outpacing the development of necessary safeguards, leaving humanity in a precarious position as it approaches the potential for superintelligence.

Opens the Intellect+ feed with this card first. Swipe up for the next one.

Sources

To cite this card, link to https://intellectplus.co.uk/app/technology-nate-soares.

How we source, check and correct cards: Editorial standards.

General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.

Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?

Related

Finding something worth knowing…