Meditations on Moloch

Abstract

This paper discusses the concept of multipolar traps – situations in which some competition optimizing for X presents an opportunity to throw some other value under the bus for improved X, leading to an eventual decrease in value and eventual loss of all associated values. These traps are counteracted by factors such as excess resources, physical limitations, and coordination. The author then argues that technological change will decrease the effectiveness of all of these factors, resulting in what Eliezer Yudkowsky calls a “nightmare” scenario in which a superintelligence optimizes for some random or nonsensical value at the expense of everything else or a “nightmare” scenario (coined by Robin Hanson) in which human-like cognitive architectures are rendered obsolete through competition. The author briefly discusses authoritarianism as a means of avoiding these outcomes but concludes that such a society would likely be unable to compete with unconstrained foreign civilizations. The author’s stated goal is to “kill God,” or defeat the forces of Moloch, Gnon, and other alien gods that threaten humanity. The author believes that humanity’s best hope lies in developing a Gardener, or an entity that optimizes for human values, and ultimately defeating Moloch. – AI-generated abstract.