Complexity, learning effects, and plausibility of vignettes in factorial surveys

Abstract

The factorial survey is a research method of that combines the advantages of survey research with the advantages of experimental designs. Respondents react to hypothetical descriptions of objects or situations (vignettes) instead of answering single-item questions. By varying each dimension of the vignettes in an experimental design, the dimensions’ impact on respondents’ judgments or decisions can be estimated accurately. Thus, the method is able to isolate the weight of single factors which are often confounded in reality. So far, only a few methodological studies address questions of validity of measurement within factorial surveys. The paper provides an overview of the use of factorial surveys in social sciences and open methodological questions. Three of these methodological problems are discussed in more detail and studied empirically using data from an online-experiment: (1) the effects of different complex situations presented to respondents, (2) learning effects due to a repeated presentation of the vignettes and (3) the effects of implausible vignettes on respondents’ reactions. According to our results all three aspects matter: A high complexity of the vignettes leads to weaker effects of single dimensions while the consistency of the judgments remains the same. Until the tenth vignette there is evidence for a learning effect: Response speed increases at a stable rate of consistent responses. Implausible combinations of vignette dimensions cause the respondents to neglect the respective dimensions and therefore lead to artificial judgments. Finally, we discuss the practical consequences of these findings.