Which Future?
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
A wide-ranging essay exploring how humanity can navigate the risks of transformative technologies, especially artificial superintelligence. Drawing on historical examples like Castle Bravo, asbestos, and nuclear weapons, the author argues that sufficiently deep understanding of reality is inherently dual-use — enabling both great benefits and catastrophic threats. The essay critiques technical AI alignment as insufficient, arguing it primarily serves market goals while accelerating dangerous capabilities. The author introduces the 'vulnerable world problem' — the question of whether reality contains simple, cheap 'recipes for ruin' — and argues ASI could discover such threats far faster than humanity can defend against them. The essay calls for developing ideas and institutions that increase the 'supply of safety' while preserving humane values, drawing parallels to fire safety, aviation safety, and nuclear arms control. It concludes with reflections on the moral choices facing scientists and technologists, using Joseph Rotblat and John Wheeler as contrasting models.
Table of contents
Which Future?Which future?AI for virus designExistential riskASI and existential riskLoss-of-control, technical alignment, and external alignmentMarket-supplied safetyHow to increase the supply of safety?The real alignment problemConclusionAcknowledgementsCitation informationFootnotesSort: