Crowdsourcing data collection from research participants recruited from online labor markets is now common in cognitive science. We review who is in the crowd and who can be reached by the average laboratory. We discuss reproducibility and review some recent methodological innovations for online experiments. We consider the design of research studies and arising ethical issues. We review how to code experiments for the web, what is known about video and audio presentation, and the measurement of reaction times. We close with comments about the high levels of experience of many participants and an emerging tragedy of the commons.

中文翻译：

---

认知科学中的众包样本

从在线劳动力市场招募的研究参与者收集众包数据现在在认知科学中很常见。我们审查谁在人群中以及普通实验室可以接触到谁。我们讨论了可重复性并回顾了一些最近的在线实验方法创新。我们考虑研究的设计和出现的伦理问题。我们回顾了如何为网络编写实验代码、视频和音频演示的已知知识以及反应时间的测量。我们最后评论了许多参与者的高水平经验和正在出现的公地悲剧。