Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.,Psychological Bulletin

当前位置： X-MOL 学术 › Psychological Bulletin › 论文详情

Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)

Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.
Psychological Bulletin ( IF 17.3 ) Pub Date : 2020-01-16 , DOI: 10.1037/bul0000220
Justin F Landy ₁ , Miaolei Liam Jia ₂ , Isabel L Ding ₃ , Domenico Viganola ₄ , Warren Tierney ₅ , Anna Dreber ₆ , Magnus Johannesson ₆ , Thomas Pfeiffer ₇ , Charles R Ebersole ₈ , Quentin F Gronau ₉ , Alexander Ly ₉ , Don van den Bergh ₉ , Maarten Marsman ₉ , Koen Derks ₁₀ , Eric-Jan Wagenmakers ₉ , Andrew Proctor ₆ , Daniel M Bartels ₁₁ , Christopher W Bauman ₁₂ , William J Brady ₁₃ , Felix Cheung ₁₄ , Andrei Cimpian ₁₃ , Simone Dohle ₁₅ , M Brent Donnellan ₁₆ , Adam Hahn ₁₅ , Michael P Hall ₁₇ , William Jiménez-Leal ₁₈ , David J Johnson ₁₉ , Richard E Lucas ₁₆ , Benoît Monin ₂₀ , Andres Montealegre ₁₈ , Elizabeth Mullen ₂₁ , Jun Pang ₂₂ , Jennifer Ray ₁₃ , Diego A Reinero ₁₃ , Jesse Reynolds ₂₃ , Walter Sowden ₁₇ , Daniel Storage ₂₄ , Runkun Su ₂₅ , Christina M Tworek ₂₆ , Jay J Van Bavel ₁₃ , Daniel Walco ₂₇ , Julian Wills ₁₃ , Xiaobing Xu ₂₈ , Kai Chi Yam ₂₉ , Xiaoyu Yang ₃₀ , William A Cunningham ₃₁ , Martin Schweinsberg ₃₂ , Molly Urwitz ₆ , The Crowdsourcing Hypothesis Tests Collaboration , Eric L Uhlmann ₃₃

Affiliation

Department of Psychology and Neuroscience, Nova Southeastern University.
Warwick Business School, University of Warwick.
Department of Marketing, National University of Singapore.
Department of Engineering, C4I and Cyber Center, George Mason University.
Kemmy Business School, University of Limerick.
Department of Economics, Stockholm School of Economics.
New Zealand Institute for Advanced Study, Massey University.
Department of Psychology, University of Virginia.
Faculty of Social and Behavioural Sciences, University of Amsterdam.
Nyenrode Business University.
Department of Marketing, Booth School of Business, University of Chicago.
Paul Merage School of Business, University of California, Irvine.
Department of Psychology, New York University.
School of Public Health, University of Hong Kong.
Social Cognition Center Cologne, University of Cologne.
Department of Psychology, Michigan State University.
Department of Psychology, University of Michigan.
Department of Psychology, University of the Andes.
Department of Sociology, University of Maryland at College Park.
Graduate School of Business and Department of Psychology, Stanford University.
School of Management, Lucas College, San José State University.
Department of Marketing, Renmin Business School, Renmin University of China.
Department of Psychology, Stanford University.
Department of Psychology, University of Denver.
NUS Business School, National University of Singapore.
HarrisX.
New York Yankees.
Hainan University.
Department of Management and Organization, National University of Singapore.
Department of Electronic Engineering, Tsinghua University.
Rotman School of Management, University of Toronto.
Department of Organizational Behavior, European School of Management and Technology.
Department of Organizational Behavior, INSEAD.

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N > 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = -0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

中文翻译：

众包假设检验：使设计选择如何影响研究结果透明。

研究结果在多大程度上受到科学家在设计研究时所做出的主观决定的影响？15个研究团队独立设计了研究，以回答与道德判断，谈判和内隐认知有关的五个原始研究问题。然后将来自2个单独的大样本（总N> 15,000）的参与者随机分配以完成每个研究的1个版本。在设计用于检验相同假设的不同材料集上，效果大小差异很大：来自不同团队的材料对5个假设中的4个在相反方向上产生了统计上显着的效果，估计值的最窄范围为d = -0.37至+ 0.26。荟萃分析和贝叶斯对结果的看法表明，总体支持2个假设，而缺乏对3个假设的支持。总体，实际上，效应大小的可变性都不归因于研究团队在设计材料方面的技能，而相当大的可变性归因于所检验的假设。在一项预测调查中，其他科学家的预测在假设范围之内和之内都与研究结果显着相关。对研究假设的众包测试有助于揭示对科学主张的经验支持的真实一致性。（PsycInfo数据库记录（c）2020 APA，保留所有权利）。在假设之内和之中。对研究假设的众包测试有助于揭示对科学主张的经验支持的真实一致性。（PsycInfo数据库记录（c）2020 APA，保留所有权利）。在假设之内和之中。对研究假设的众包测试有助于揭示对科学主张的经验支持的真实一致性。（PsycInfo数据库记录（c）2020 APA，保留所有权利）。

更新日期：2020-01-16

点击分享查看原文

点击收藏

公开下载

阅读更多本刊最新论文