Research Repository

Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

Schweinsberg, Martin and Feldman, Michael and Staub, Nicola and van den Akker, Olmo R and van Aert, Robbie CM and van Assen, Marcel ALM and Liu, Yang and Althoff, Tim and Heer, Jeffrey and Kale, Alex and Mohamed, Zainab and Amireh, Hashem and Prasad, Vaishali Venkatesh and Bernstein, Abraham and Robinson, Emily and Snellman, Kaisa and Sommer, S Amy and Otner, Sarah MG and Robinson, David and Madan, Nikhil and Silberzahn, Raphael and Goldstein, Pavel and Tierney, Warren and Murase, Toshio and Mandl, Benjamin and Viganola, Domenico and Strobl, Carolin and Schaumans, Catherine BC and Kelchtermans, Stijn and Naseeb, Chan and Garrison, S Mason and Yarkoni, Tal and Chan, CS Richard and Prestone, Adie and Alaburda, Paulius and Albers, Casper and Alspaugh, Sara and Alstott, Jeff and Nelson, Andrew A and Ariño de la Rubia, Eduardo and Arzi, Adbi and Bahník, Štěpán and Baik, Jason and Balling, Laura Winther and Banker, Sachin and Baranger, David AA and Barr, Dale J and Barros-Rivera, Brenda and Bauer, Matt and Blaise, Enuh and Boelen, Lisa and Bohle Carbonell, Katerina and Briers, Robert A and Burkhard, Oliver and Canela, Miguel-Angel and Castrillo, Laura and Catlett, Timothy and Chen, Olivia and Clark, Michael and Cohn, Brent and Coppock, Alex and Cugueró-Escofet, Natàlia and Curran, Paul G and Cyrus-Lai, Wilson and Dai, David and Dalla Riva, Giulio Valentino and Danielsson, Henrik and de F S M Russo, Rosaria and de Silva, Niko and Derungs, Curdin and Dondelinger, Frank and Duarte de Souza, Carolina and Dube, B Tyson and Dubova, Marina and Dunn, Ben Mark and Edelsbrunner, Peter Adriaan and Finley, Sara and Fox, Nick and Gnambs, Timo and Gong, Yuanyuan and Grand, Erin and Greenawalt, Brandon and Han, Dan and Hanel, Paul HP and Hong, Antony B and Hood, David and Hsueh, Justin and Huang, Lilian and Hui, Kent N and Hultman, Keith A and Javaid, Azka and Jiang, Lily Ji and Jong, Jonathan and Kamdar, Jash and Kane, David and Kappler, Gregor and Kaszubowski, Erikson and Kavanagh, Christopher M and Khabsa, Madian and Kleinberg, Bennett and Kouros, Jens and Krause, Heather and Krypotos, Angelos-Miltiadis and Lavbič, Dejan and Lee, Rui Ling and Leffel, Timothy and Lim, Wei Yang and Liverani, Silvia and Loh, Bianca and Lønsmann, Dorte and Low, Jia Wei and Lu, Alton and MacDonald, Kyle and Madan, Christopher R and Madsen, Lasse Hjorth and Maimone, Christina and Mangold, Alexandra and Marshall, Adrienne and Matskewich, Helena Ester and Mavon, Kimia and McLain, Katherine L and McNamara, Amelia A and McNeill, Mhairi and Mertens, Ulf and Miller, David and Moore, Ben and Moore, Andrew and Nantz, Eric and Nasrullah, Ziauddin and Nejkovic, Valentina and Nell, Colleen S and Nelson, Andrew Arthur and Nilsonne, Gustav and Nolan, Rory and O’Brien, Christopher E and O’Neill, Patrick and O’Shea, Kieran and Olita, Toto and Otterbacher, Jahna and Palsetia, Diana and Pereira, Bianca and Pozdniakov, Ivan and Protzko, John and Reyt, Jean-Nicolas and Riddle, Travis and Ridhwan Omar Ali, Amal Akmal and Ropovik, Ivan and Rosenberg, Joshua M and Rothen, Stephane and Schulte-Mecklenbeck, Michael and Sharma, Nirek and Shotwell, Gordon and Skarzynski, Martin and Stedden, William and Stodden, Victoria and Stoffel, Martin A and Stoltzman, Scott and Subbaiah, Subashini and Tatman, Rachael and Thibodeau, Paul H and Tomkins, Sabina and Valdivia, Ana and Druijff-van de Woestijne, Gerrieke B and Viana, Laura and Villesèche, Florence and Wadsworth, W Duncan and Wanders, Florian and Watts, Krista and Wells, Jason D and Whelpley, Christopher E and Won, Andy and Wu, Lawrence and Yip, Arthur and Youngflesh, Casey and Yu, Ju-Chi and Zandian, Arash and Zhang, Leilei and Zibman, Chava and Uhlmann, Eric Luis (2021) 'Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis.' Organizational Behavior and Human Decision Processes. ISSN 0749-5978

[img]
Preview
Text
1-s2.0-S0749597821000200-main.pdf - Published Version
Available under License Creative Commons Attribution.

Download (3MB) | Preview

Abstract

In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for organizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed.

Item Type: Article
Uncontrolled Keywords: crowdsourcing data analysis; scientific transparency; research reliability; scientific robustness; researcher degrees of freedom; analysis-contingent results
Divisions: Faculty of Science and Health > Psychology, Department of
Depositing User: Elements
Date Deposited: 24 Feb 2021 15:22
Last Modified: 02 Jul 2021 13:28
URI: http://repository.essex.ac.uk/id/eprint/29938

Actions (login required)

View Item View Item