My understanding of Paul Christiano's iterated amplification AI safety research agenda

Abstract

Paul Christiano’s Iterated Amplification (IDA) research agenda aims to find a safe and powerful version of self-play methods for training Artificial Intelligence (AI) systems. The agenda proposes using a slow, safe method to scale up an AI’s capabilities, followed by distilling this knowledge into a faster, slightly weaker AI. This process can then be iterated until a fast and powerful AI is achieved. IDA proposes that the process of scaling up an AI’s capabilities can be achieved through factored cognition, which involves giving a weak and safe AI access to other weak and safe AIs which they can ask questions to solve more difficult tasks. While IDA could be successful in developing a universally competent AI with safe and strong capabilities, several potential failure modes exist. HCH, an idealized model of factored cognition, could turn out to be not powerful enough, not corrigible, or not translatable to real-world AI systems. Additionally, the differential competence problem may arise, where HCH favors some skills over others, leading to bad combinations of abilities. IDA also faces criticism regarding the concept of corrigibility, with some critics arguing that a clear theoretical understanding of corrigibility is necessary for it to work in AI. – AI-generated abstract.