What are participants doing while filling in an online questionnaire : a paradata collection tool and an empirical study

Abstract

The use of online questionnaires is rapidly increasing. Contrary to manifold advantages, not much is known about user behavior that can be measured outside the boundaries set by standard web technologies like HTML form elements. To show how the lack of knowledge about the user setting in web studies can be accounted for, we present a tool called UserActionTracer, with which it is possible to collect more behavior information than with any other paradata gathering tool, in order to (1) gather additional data unobtrusively from the process of answering questions and (2) to visualize individual user behavior on web pages. In an empirical study on a large web sample (N = 1046) we observed and analysed online behaviors (e.g., clicking through). We found that only 10.5% of participants showed more than five single behaviors with highly negative influence on data quality in the whole online questionnaire (out of 132 possible single behavior judgments). Furthermore, results were validated by comparison with data from online address books. With the UserActionTracer it is possible to gain further insight into the process of answering online questionnaires.