The nature of self-improving artificial intelligence

Abstract

Self-improving systems are a promising new approach to developing ar- tificial intelligence. But will their behavior be predictable? Can we be sure that they will behave as we intended even after many generations of self- improvement? This paper presents a framework for answering questions like these. It shows that self-improvement causes systems to converge on an architecture that arises from von Neumanns foundational work on microe- conomics. Self-improvement causes systems to allocate their physical and computational resources according to a universal principle. It also causes systems to exhibit four natural drives: 1) efficiency, 2) self-preservation, 3) resource acquisition, and 4) creativity. Unbridled, these drives lead to both desirable and undesirable behaviors. The efficiency drive leads to algorithm optimization, data compression, atomically precise physical structures, re- versible computation, adiabatic physical action, and the virtualization of the physical. It also governs a systems choice of memories, theorems, lan- guage, and logic. The self-preservation drive leads to defensive strategies such as energy encryption for hiding resources and promotes replication and game theoretic modeling. The resource acquisition drive leads to a va- riety of competitive behaviors and promotes rapid physical expansion and imperialism. The creativity drive leads to the development of new concepts, algorithms, theorems, devices, and processes. The best of these traits could usher in a new era of peace and prosperity; the worst are characteristic of human psychopaths and could bring widespread destruction. How can we ensure that this technology acts in alignment with our values? We have leverage both in designing the initial systems and in creating the social con- text within which they operate. But we must have clarity about the future we wish to create. We need not just a logical understanding of the technology but a deep sense of the values we cherish most. With both logic and inspi- ration we can work toward building a technology that empowers the human spirit rather than diminishing it.