Convergence and Compromise: Will Society Aim for Good Futures?

Abstract

This work examines the likelihood of achieving a “mostly-great future,” given the premise that such an outcome is a narrow target. The possibility of widespread, accurate, and motivational (WAM) moral convergence, where most actors identify and pursue the good de dicto, is considered unlikely. Arguments for WAM-convergence based on current moral agreement or past moral progress are found to be weak, as present consensus is largely instrumental and historical progress may be contingent or insufficient for future ethical challenges. Even with post-AGI advancements like superintelligent advice and material abundance, fundamental meta-ethical problems persist: under moral realism, the correct view may be too alien to be motivating, while under anti-realism, diverse starting points are unlikely to converge on the highly specific values required. A more plausible, though fraught, pathway is partial moral convergence combined with trade and compromise. In this scenario, a minority motivated by the good could bargain to secure highly valuable outcomes. However, this mechanism is vulnerable to value-destroying threats and extortion, which could negate or reverse any gains. Finally, achieving a great future as an unintended byproduct of other motives, like self-interest, is improbable due to the specificity of the target. – AI-generated abstract.