[AN #156]: the Scaling Hypothesis: a Plan for Building AGI

Abstract

The scaling hypothesis, which proposes that intelligence or human-level prediction comes from training larger NNs on more diverse datasets, is supported by scaling laws and the recent performance of various large-scale models like GPT-3, AlphaStar, and Image GPT. Despite this, most researchers seem to ignore the scaling hypothesis. The training objective used in these large models, such as predicting the next word, may not be as important as the dataset used to train it. Hence, reward functions often employed to guide the actions of an AI model, may not be as indicative of properties in the resultant trained model. – AI-generated abstract.