Understanding question-reading deviations: Implications for monitoring interviewers, questionnaire design, and data quality

Chapter 1 examines the accuracy and utility of using paradata to detect interviewer question-reading deviations. Using timestamps and behavior coded data from interviewer recordings, I explore different methods (i.e., different rates and ranges of reading pace, standard deviations, and model-based methods) for constructing question administration timing thresholds (QATT) and compare them to the behavior coded data to determine the accuracy and utility of each method to detect minor and major deviations. Results show that using a reading rate of 4 words per second (WPS)to create upper and lower QATTs has the highest overall accuracy (87.1%) and the most utility for correctly identifying interviews with and without major deviations. Chapter 2 examines the impact question characteristics have on question-reading deviations in face-to-face interviews. To evaluate this, questions from theInnovation Panel(IP) Wave 3 were coded on the following dimensions: structure, content, and the presence of interviewer aids, resulting in 19 question characteristics. Results show that of the 19 question characteristics examined, 16 are significantly associated with major question-reading deviations. The question characteristics that have the highest odds of major deviations are questions that have definitions or examples (6.404), questions that have response options read in the question text (4.133), and demographic questions (2.421). Chapter 3 examines the impact of question-reading deviations on data quality. Several measures are used to assess data quality, including item non-response and differences in response distributions for questions that are read verbatim (or have minor deviations) and questions that have major deviations. The results show that major question-reading deviations are only significantly associated with question timing; changed wording has a significant negative association with question timing. The other data quality indicators (i.e., Don't Know and distribution of means) showed no significant effect from major question-wording deviations.