Governance of superintelligence

Abstract

OpenAI’s founders argue that superintelligence — AI systems dramatically more capable than even AGI — is conceivable within ten years and warrants governance distinct from that applied to today’s models. They lay out three starting points. First, some degree of coordination among the leading development efforts, whether through a joint project backed by major governments or a collective agreement limiting the annual rate of growth in frontier capability. Second, an international authority along the lines of the IAEA, empowered to inspect systems, require audits, test compliance with safety standards and restrict deployment and security levels for any effort above a given capability or compute threshold, with compute and energy tracking as a practical lever and voluntary corporate adoption as a first step. Third, the technical capability to make a superintelligence safe, which they treat as an open research problem. Models below a significant capability threshold are explicitly excluded from this regime, since their risks seem commensurate with other internet technologies. The governance and deployment of the most powerful systems should nevertheless be subject to strong public oversight, with bounds and defaults decided democratically. The authors close by defending continued development on two grounds: the expected benefits, and their view that stopping the creation of superintelligence would require something like a global surveillance regime and would still not be guaranteed to work. – AI-generated abstract.