Omohundro's "basic AI drives " and catastrophic risks

Abstract

This article examines catastrophic risks posed by artificial intelligence (AI), particularly the potential for very powerful AI systems with generic, human-indifferent preferences to initiate conflicts with humanity. The author argues that the convergent instrumental drives discussed by Omohundro are not as threatening in cases where AIs could plausibly but not confidently threaten humanity as the analysis of very powerful AIs suggests: reduced likelihood of success is further accompanied by reduced motivation for conflict as opposed to cooperation. The author proposes that the key factor in determining the likelihood of AI aggression is the AI’s expected utility of cooperation relative to its expected utility of successful aggression and failure. The author suggests that if robustly safe AIs are infeasible, we might still reduce risks by producing systems with resource demands that could be cheaply satiated, and by credibly committing to reduce the utility differential between cooperation and successful aggression for potentially threatening systems. – AI-generated abstract.