From GPT-4 to AGI: Counting the OOMs

Abstract

Deep learning has exhibited rapid progress over the past decade, with models advancing from basic image recognition to complex problem-solving and exceeding human performance on various benchmarks. This progress is driven by scaling compute, algorithmic efficiencies, and “unhobbling” gains that unlock latent model capabilities. From GPT-2 to GPT-4, a period of roughly four years, effective compute increased by 4.5–6 orders of magnitude, coupled with significant unhobbling advancements like RLHF and chain-of-thought prompting. This led to a qualitative leap in performance comparable to the cognitive development from preschooler to high-schooler. Projecting these trends forward, another similar jump is anticipated by 2027, potentially driven by a further 3–6 order of magnitude increase in effective compute and continued unhobbling, such as improvements in long-term memory, tool use, and test-time compute. This could lead to the development of artificial general intelligence (AGI) capable of performing tasks comparable to expert humans and automating a wide range of cognitive jobs, potentially triggering rapid further advancements due to self-improving AI. However, data limitations and the need for algorithmic breakthroughs to overcome them pose a significant source of uncertainty. – AI-generated abstract.