Methodological effects of factorial surveys in population samples

Abstract

The factorial survey is a method of attitude research that combines the advantages of survey research and experimental designs. Respondents react to hypothetical descriptions (vignettes) instead of answering single-item questions. By varying each dimen¬sion its impact on respondents' judgments can be estimated. As the descriptions are based on a number of dimensions and respondents have to evaluate between 10 and 20 descriptions factorial surveys are demanding with regard to cognitive ability and the capability to information processing. There is little empirical knowledge if and to what extend this higher complexity causes methodological artifacts. The problem becomes apparent when factorial surveys are conducted in population samples where the required abilities are heterogeneous. Using two sources of information – interviewer evaluations and respondent's behavior – this paper asks for the effects of educational background and age on the comprehension and performance within a factorial survey conducted in a population sample of 1066 randomly selected German inhabitants. Analyses of interviewer evaluations show that: (1) older respondents have more comprehension problems, (2) there are almost no differences in response time but in consistency and information usage between educational and age groups, and (3) respondents took fewer dimensions into account during the survey which is interpreted as a fatigue effect. The results demonstrate that factorial surveys are applicable in population surveys but should be used with a smaller number of vignettes per respondent than in student samples.