Arms control and intelligence explosions

Abstract

A number of commentators have argued that some time in the 21st century humanity will develop generally intelligent software programs at least as capable as skilled humans, whether designed ab initio or as emulations of human brains, and that such entities will launch an extremely rapid technological transformation as they design their own suc- cessors. The speed of such a “Singularity” or “intelligence explosion” would be so great that biological humans would lack time for extensive deliberation regarding or super- vision of the process. Several authors have called for regulation to retard the pace of advancement in this field, allowing more time to ensure that any intelligent machines are safe and broadly beneficial, while various proponents of the Singularity hypothesis have replied that such attempts will fail because of competition between regulatory ju- risdictions, sometimes making analogies to the failures of nuclear counter-proliferation efforts. This paper discusses some key considerations that distinguish the case of sapient software programs from the historical experience with nuclear weapons technology