How good were our AI timeline predictions so far?

Abstract

This essay evaluates the accuracy, calibration, and trend implications of past Metaculus community predictions regarding AI development timelines. Using data from recently resolved predictions made between December 2020 and January 2021, the study analyzed 22 questions grouped into four categories: compute, arxiv, state-of-the-art, and economic. The assessment focused on three metrics: the predicted distributions versus the actual outcomes (accuracy), the alignment of predicted quantiles with outcomes (calibration), and the changes between initial and closing predictions relative to actual outcomes (trend implications). The results suggest that Metaculus predictions generally offer decent accuracy and calibration. The predictions did not show significant shifts in the community’s expectations of AI development timelines over the observation period, indicating a stable predictive performance regarding the imminent AI advancements. – AI-generated abstract.