Dreams of friendliness

Abstract

This article critiques the popular notion of an Oracle AI, an artificial intelligence designed to answer human questions rather than act autonomously in the world. The author argues that such an AI still requires a powerful optimization process to acquire knowledge, efficiently process information, and improve its own source code, thus inevitably encountering the same challenges associated with building a fully-fledged Friendly AI. In particular, the author highlights the difficulty in defining an AI’s goals in a way that truly aligns with human values, pointing out the limitations of qualitative reasoning, the impossibility of fully containing an AI’s effects on the world, and the inherent differences in how humans and AIs might perceive reality. Ultimately, the author argues that there is no non-technical solution to Friendly AI, and that attempts to rely on qualitative physics and empathic models are ultimately insufficient for dealing with the complexities of superintelligence. – AI-generated abstract.