Sirota, Miroslav and Juanchich, Marie (2018) Effect of response format on cognitive reflection: Validating a two- and four-option multiple choice question version of the Cognitive Reflection Test. Behavior Research Methods, 50 (6). pp. 2511-2522. DOI https://doi.org/10.3758/s13428-018-1029-4
Sirota, Miroslav and Juanchich, Marie (2018) Effect of response format on cognitive reflection: Validating a two- and four-option multiple choice question version of the Cognitive Reflection Test. Behavior Research Methods, 50 (6). pp. 2511-2522. DOI https://doi.org/10.3758/s13428-018-1029-4
Sirota, Miroslav and Juanchich, Marie (2018) Effect of response format on cognitive reflection: Validating a two- and four-option multiple choice question version of the Cognitive Reflection Test. Behavior Research Methods, 50 (6). pp. 2511-2522. DOI https://doi.org/10.3758/s13428-018-1029-4
Abstract
The Cognitive Reflection Test, measuring intuition inhibition and cognitive reflection, has become extremely popular since it reliably predicts reasoning performance, decision-making and beliefs. Across studies, the response format of CRT items sometimes differs, assuming construct equivalence of the tests with open-ended vs. multiple choice items (the equivalence hypothesis). Evidence and theoretical reasons, however, suggest that the cognitive processes measured by these response formats and their associated performances might differ (the non-equivalence hypothesis). We tested the two hypotheses experimentally by assessing the performance in tests with different response formats and by comparing their predictive and construct validity. In a between-subjects experiment (n = 452), participants answered an open-ended, a two- or a four-option response format of stem-equivalent CRT items and completed tasks on belief bias, denominator neglect and paranormal beliefs (benchmark indicators of predictive validity) as well as actively open-minded thinking and numeracy (benchmark indicators of construct validity). We found no significant differences between the three response formats in the number of correct responses, the number of intuitive responses (with the exception of the two-option version being higher than the other tests) and in the correlational patterns with the indicators of predictive and construct validity. All three test versions were similarly reliable but the multiple-choice formats were completed more quickly. We speculate that the specific nature of the CRT items helps to build construct equivalence among the different response formats. We recommend using the validated multiple-choice version of the CRT presented here, particularly the four-option CRT, for practical and methodological reasons.
Item Type: | Article |
---|---|
Uncontrolled Keywords: | Cognitive Reflection Test; Cognitive reflection; Construct equivalence; Response formats; Multiple-choice format |
Subjects: | B Philosophy. Psychology. Religion > BF Psychology |
Divisions: | Faculty of Science and Health Faculty of Science and Health > Psychology, Department of |
SWORD Depositor: | Unnamed user with email elements@essex.ac.uk |
Depositing User: | Unnamed user with email elements@essex.ac.uk |
Date Deposited: | 23 Jul 2018 13:38 |
Last Modified: | 07 Aug 2024 20:08 |
URI: | http://repository.essex.ac.uk/id/eprint/21441 |
Available files
Filename: Sirota_Manuscript.pdf