works
Eliezer Yudkowsky The Bottom Line online The process of belief formation determines the evidential value of a conclusion, distinct from the justifications presented afterward. This is illustrated through a parable involving two boxes, one containing a diamond, with uncertain signs indicating its location. A “clever arguer” is hired to advocate for one box; they first commit to the conclusion that their client’s box holds the diamond and subsequently select only supporting arguments. The conclusion written by this arguer is causally entangled only with the hiring process, not the actual location of the diamond. Conversely, a “curious inquirer” first examines all available evidence impartially before deriving a probabilistic conclusion. This inquirer’s conclusion is entangled with the signs and portents related to the boxes. Applied to reasoning, the effectiveness of one’s conclusions depends on the underlying “algorithm” that determines the “bottom line.” If a conclusion is fixed beforehand due to preference or bias, the subsequent arguments gathered to support it do not change the epistemic status determined by the initial, potentially flawed, decision process. This framework serves as a caution for self-reflection on one’s own reasoning pathways, rather than primarily as a method for critiquing others. – AI-generated abstract.

The Bottom Line

Eliezer Yudkowsky

LessWrong, September 28, 2007

Abstract

The process of belief formation determines the evidential value of a conclusion, distinct from the justifications presented afterward. This is illustrated through a parable involving two boxes, one containing a diamond, with uncertain signs indicating its location. A “clever arguer” is hired to advocate for one box; they first commit to the conclusion that their client’s box holds the diamond and subsequently select only supporting arguments. The conclusion written by this arguer is causally entangled only with the hiring process, not the actual location of the diamond. Conversely, a “curious inquirer” first examines all available evidence impartially before deriving a probabilistic conclusion. This inquirer’s conclusion is entangled with the signs and portents related to the boxes. Applied to reasoning, the effectiveness of one’s conclusions depends on the underlying “algorithm” that determines the “bottom line.” If a conclusion is fixed beforehand due to preference or bias, the subsequent arguments gathered to support it do not change the epistemic status determined by the initial, potentially flawed, decision process. This framework serves as a caution for self-reflection on one’s own reasoning pathways, rather than primarily as a method for critiquing others. – AI-generated abstract.

PDF

First page of PDF