Comparing the validity of trait estimates from the multidimensional forced-choice format and the rating scale format

Abstract

The multidimensional forced-choice (MFC) format has been proposed as an alternative to rating scales (RS) that may be less susceptible to response biases. The goal of this study was to compare the validity of trait estimates from the MFC and the RS format when using normative scoring for both formats. We focused on construct validity and criterion-related validity. In addition, we investigated test-retest reliability over a period of six months. Participants were randomly assigned the MFC (N = 593) or the RS (N = 622) version of the Big Five Triplets. In addition to self-ratings on the Big Five Triplets and other personality questionnaires and criteria, we also obtained other-ratings (N = 770) for the Big Five Triplets. The Big Five in the Big Five Triplets corresponded well with the Big Five in the Big Five Inventory except for agreeableness in the MFC version. The majority of the construct validity coefficients differed between the MFC and the RS version, whereas criterion-related validities were very similar. The self- and other-rated Big Five Triplets showed higher correlations in the MFC format than in the RS format. The reliability of trait estimates on the Big Five and test-retest reliabilities were lower for MFC compared to RS. For the MFC format to be able to replace the RS format, more research on how to obtain ideal constellations of items that are matched in their desirability is needed.