The human factor is prevalent in empirical software engineering research. However, human studies often do not use the full potential of analysis methods by combining analysis of individual tasks and participants with an analysis that aggregates results over tasks and/or participants. This may hide interesting insights of tasks and participants and may lead to false conclusions by overrating or underrating single-task or participant performance. We show that studying multiple levels of aggregation of individual tasks and participants allows researchers to have both insights from individual variations as well as generalized, reliable conclusions based on aggregated data. Our literature survey revealed that most human studies perform either a fully aggregated analysis or an analysis of individual tasks. To show that there is important, non-trivial variation when including human participants, we reanalyze 12 published empirical studies, thereby changing the conclusions or making them more nuanced. Moreover, we demonstrate the effects of different aggregation levels by answering a novel research question on published sets of fMRI data. We show that when more data are aggregated, the results become more accurate. This proposed technique can help researchers to find a sweet spot in the tradeoff between cost of a study and reliability of conclusions.

中文翻译：

---

掌握人类研究的变化

人为因素在经验软件工程研究中很普遍。然而，人类研究通常不会通过将单个任务和参与者的分析与汇总任务和/或参与者的结果的分析相结合来充分利用分析方法的潜力。这可能会隐藏任务和参与者的有趣见解，并可能通过高估或低估单个任务或参与者的表现而导致错误的结论。我们表明，研究单个任务和参与者的多层次聚合使研究人员既可以从个体差异中获得见解，也可以基于聚合数据获得普遍、可靠的结论。我们的文献调查显示，大多数人类研究要么进行完全汇总的分析，要么对单个任务进行分析。为了表明有重要意义，当包括人类参与者时，我们重新分析了 12 项已发表的实证研究，从而改变了结论或使它们更加细致入微。此外，我们通过回答关于已发布的 fMRI 数据集的新研究问题来证明不同聚合水平的影响。我们表明，当聚合更多数据时，结果会变得更加准确。这种提议的技术可以帮助研究人员在研究成本和结论可靠性之间找到最佳平衡点。结果变得更加准确。这种提议的技术可以帮助研究人员在研究成本和结论可靠性之间找到最佳平衡点。结果变得更加准确。这种提议的技术可以帮助研究人员在研究成本和结论可靠性之间找到最佳平衡点。