An Accuracy Argument in Favor of Ranking Theory

Abstract

Fitelson and McCarthy (2014) have proposed an accuracy measure for confidence orders which favors probability measures and Dempster-Shafer belief functions as accounts of degrees of belief and excludes ranking functions. Their accuracy measure only penalizes mistakes in confidence comparisons. We propose an alternative accuracy measure that also rewards correct confidence comparisons. Thus we conform to both of William James’ maxims: “Believe truth! Shun error!” We combine the two maxims, penalties and rewards, into one criterion that we call prioritized accuracy optimization (PAO). That is, PAO punishes wrong comparisons (preferring the false to the true) and rewards right comparisons (preferring the true to the false). And it requires to prioritize being right und avoiding to be wrong in a specific way. Thus PAO is both, a scoring rule and a decision rule. It turns out that precisely confidence orders representable by two-sided ranking functions satisfy PAO. The point is not to argue that PAO is the better accuracy goal. The point is only that ranking theory can also be supported by accuracy considerations. Thus, those considerations by themselves cannot decide about rational formats for degrees of belief, but are part and parcel of an overall normative assessment of those formats.