How accurate are open phil's predictions?

Abstract

An internal Open Philanthropy investigation into 2,850 predictions made by grant investigators from 2022 to 2023 reveals an overall modest accuracy marked by overconfidence in the highest confidence bin (predictions made with 90% certainty). The examination further suggests that calibration alone contributes more to accuracy than resolution, with the implication that greater effort is needed to improve the latter at the expense of the former. The study recommends increasing the number of predictions per grant, though it notes that more time investment in each prediction may be necessary to improve resolution, which it considers not to be worth the effort. The investigators also caution against certain caveats and biases present in their data (e.g., selection effects, scoring issues, and exploratory analyses), but ultimately present their work as a useful means of learning how to improve their grantmaking effectiveness and accountability. – AI-generated abstract.