Is power-seeking AI an existential risk?

Abstract

This writing introduces a visual model named “skyscrapers and madmen” to explicate expected utility maximization (EUM), where the skyscrapers represent outcomes of actions, and the height of a skyscraper indicates the utility of an action’s outcome. It explains that EUM often results in predictably losing situations, where the probabilities of success are low but the potential benefits are high. The article argues that these predictably losing situations can be justified by the fact that they maximize expected utility in the long run. However, the article also critiques some arguments in favor of EUM, asserting that they are either insufficient or incomplete. – AI-generated abstract.