Method comparisons are essential to provide recommendations and guidance for applied researchers, who often have to choose from a plethora of available approaches. While many comparisons exist in the literature, these are often not neutral but favour a novel method. Apart from the choice of design and a proper reporting of the findings, there are different approaches concerning the underlying data for such method comparison studies. Most manuscripts on statistical methodology rely on simulation studies and provide a single real-world data set as an example to motivate and illustrate the methodology investigated. In the context of supervised learning, in contrast, methods are often evaluated using so-called benchmarking data sets, i.e. real-world data that serve as gold standard in the community. Simulation studies, on the other hand, are much less common in this context. The aim of this paper is to investigate differences and similarities between these approaches, to discuss their advantages and disadvantages and ultimately to develop new approaches to the evaluation of methods picking the best of both worlds. To this aim, we borrow ideas from different contexts such as mixed methods research and Clinical Scenario Evaluation.

中文翻译：

---

关于基准数据集和模拟在方法比较研究中的作用

方法比较对于为应用研究人员提供建议和指导至关重要，他们通常必须从大量可用方法中进行选择。虽然文献中存在许多比较，但这些比较通常不是中性的，而是有利于一种新颖的方法。除了选择设计和适当报告结果外，对于此类方法比较研究的基础数据，还有不同的方法。大多数关于统计方法的手稿都依赖于模拟研究，并提供一个单一的真实世界数据集作为示例来激发和说明所研究的方法。相比之下，在监督学习的背景下，通常使用所谓的基准数据集来评估方法，即在社区中充当黄金标准的真实数据。另一方面，模拟研究，在这种情况下不太常见。本文的目的是调查这些方法之间的差异和相似之处，讨论它们的优缺点，并最终开发出新的方法来评估两全其美的方法。为此，我们借鉴了不同背景下的想法，例如混合方法研究和临床情景评估。