Research Repository

Understanding question-reading deviations: Implications for monitoring interviewers, questionnaire design, and data quality

Kelley, Jennifer (2021) Understanding question-reading deviations: Implications for monitoring interviewers, questionnaire design, and data quality. PhD thesis, University of Essex.

Kelley_Jennifer_PhDThesis Final.pdf

Download (1MB) | Preview


Chapter 1 examines the accuracy and utility of using paradata to detect interviewer question-reading deviations. Using timestamps and behavior coded data from interviewer recordings, I explore different methods (i.e., different rates and ranges of reading pace, standard deviations, and model-based methods) for constructing question administration timing thresholds (QATT) and compare them to the behavior coded data to determine the accuracy and utility of each method to detect minor and major deviations. Results show that using a reading rate of 4 words per second (WPS)to create upper and lower QATTs has the highest overall accuracy (87.1%) and the most utility for correctly identifying interviews with and without major deviations. Chapter 2 examines the impact question characteristics have on question-reading deviations in face-to-face interviews. To evaluate this, questions from theInnovation Panel(IP) Wave 3 were coded on the following dimensions: structure, content, and the presence of interviewer aids, resulting in 19 question characteristics. Results show that of the 19 question characteristics examined, 16 are significantly associated with major question-reading deviations. The question characteristics that have the highest odds of major deviations are questions that have definitions or examples (6.404), questions that have response options read in the question text (4.133), and demographic questions (2.421). Chapter 3 examines the impact of question-reading deviations on data quality. Several measures are used to assess data quality, including item non-response and differences in response distributions for questions that are read verbatim (or have minor deviations) and questions that have major deviations. The results show that major question-reading deviations are only significantly associated with question timing; changed wording has a significant negative association with question timing. The other data quality indicators (i.e., Don't Know and distribution of means) showed no significant effect from major question-wording deviations.

Item Type: Thesis (PhD)
Subjects: H Social Sciences > H Social Sciences (General)
Divisions: Faculty of Social Sciences > Institute for Social and Economic Research
Depositing User: Jennifer Kelley
Date Deposited: 13 May 2021 12:13
Last Modified: 13 May 2021 12:13

Actions (login required)

View Item View Item