Holden Karnofsky on how AIs might take over even if they're no smarter than humans, and his four-part playbook for AI risk

Abstract

Holden Karnofsky argues that humanity should be concerned not only with the possibility of superintelligent AI but also with the prospect of an AI “population explosion”, where large numbers of human-level AI systems could emerge and outpace human decision-making capabilities. Karnofsky further outlines four critical interventions that humanity must focus on to navigate this transition: (1) AI alignment research, (2) standards and monitoring for dangerous AI capabilities, (3) prioritizing AI safety in the development of successful and influential AI labs, and (4) enhancing information security to prevent malicious actors from stealing AI systems. He argues that these interventions are more urgent than ever due to the rapid pace of progress in AI, which could lead to transformative changes in society within a short time span. Karnofsky also criticizes a narrow focus on “impartial expected welfare maximisation” as a framework for ethics, suggesting that it is not realistic and can lead to implausible consequences. – AI-generated abstract