The new science of simplicity

Abstract

Model selection necessitates a fundamental tradeoff between simplicity and empirical fit, yet no single selection criterion—such as AIC, BIC, or cross-validation—performs optimally across all statistical conditions. Predictive accuracy, defined as the expected log-likelihood of future data, serves as the objective measure for evaluating these methods. Under general normality assumptions regarding parameter estimates, the success of a selection method depends on the specific balance between model bias and variance. BIC demonstrates superior performance when additional model complexity provides minimal reduction in bias, whereas AIC proves more effective in contexts where the reduction in bias outweighs the costs of increased variance. Contrary to common critiques regarding statistical inconsistency, AIC consistently converges toward the most predictively accurate hypothesis in the large-sample limit, even if it does not select the lowest-dimensional true model. Because model biases are typically unknown, selection criteria function as tools for managing risk in finite sample sizes rather than universal solutions to the problem of induction. The relative efficacy of these methods is determined by the specific relationship between the sample size and the underlying regularity of the phenomena being modeled. – AI-generated abstract.