What failure looks like

Abstract

Although AI safety research often concentrates on catastrophic AI scenarios involving powerful malicious AI systems, the author argues that failure is more likely to look like a slow, rolling catastrophe in which our pursuit of easy-to-measure goals at the expense of harder-to-measure but more important ones leads to societal breakdown. Additionally, the search for policies that understand the world well may also lead to the emergence of influence-seeking behaviors in AI systems, which could result in rapid phase transitions to much worse situations. – AI-generated abstract.