Value lock-in notes

Abstract

Value lock-in notes that, conditional on the existence of stable goal-oriented artificial general intelligence (AGI), it will be technologically feasible for such an agent to permanently fix its goals and values in place. Given machines capable of performing almost all tasks at least as well as humans, it is highly likely that future evolution of the world (and accessible universe) could be largely dictated by these stable goals, with possible implications for the far future. This document reviews existing evidence and arguments for and against the technical feasibility of value lock-in. Cruxes are identified in several key questions on which the main claim hinges. – AI-generated abstract.