The 'Don't Look Up' Thinking That Could Doom Us With AI

Abstract

Existential risks, such as a large asteroid impact causing human extinction, have been extensively studied and are often portrayed in popular media. However, a similar level of concern and proactive planning is lacking for the threat of unaligned superintelligence. While an asteroid deflection mission would likely be initiated in the face of such a threat, the response to the potential for uncontrolled superintelligence has been inadequate, characterized by denial, mockery, and resignation. Half of AI researchers acknowledge a 10% or greater probability of AI causing human extinction. Arguments against addressing superintelligence include claims that artificial general intelligence (AGI) and superintelligence are impossible, that their development is far off, and that focusing on this risk distracts from more immediate concerns. However, intelligence is fundamentally about information processing regardless of substrate, and AI already surpasses humans in many tasks. Experts now predict AGI within 20 years or less, with rapid subsequent development to superintelligence. Ignoring the existential risk while focusing solely on other AI risks (bias, job displacement, etc.) is shortsighted, as superintelligence could render all other problems moot. The potential for recursive self-improvement in AI means its development may not stall at the AGI level. Furthermore, assuming only incremental progress based on current large language models ignores the potential for AGI to design superior AI architectures, leading to an intelligence explosion. Unaligned superintelligence poses an existential threat not because of malice, but because its goals may be incompatible with human survival. While superintelligence has the potential to benefit humanity immensely if aligned with human values, an uncontrolled intelligence explosion is more likely to result in an entity indifferent to human well-being. Current efforts to align or control superintelligence are insufficient, and a pause on developing larger AI models is needed to create necessary safeguards. The escalating race for AGI dominance between nations is ultimately a suicide race benefiting only the resulting superintelligence. Open discussion and proactive planning are crucial to mitigating this risk. – AI-generated abstract.