Reproducibility: the extent to which consistent results are obtained when an experiment is repeated, is important as a means to validate experimental results, promote integrity of research, and accelerate follow up work. Commitment to artifact reviewing and badging seeks to promote reproducibility and rank the quality of submitted artifacts. However, as illustrated in this issue, the current badging scheme, with its focus upon an artifact being reusable, may not identify limitations of architecture, implementation, or evaluation. We propose that to improve the insight into artifact reproducibility, the depth and nature of artifact evaluation must move beyond simply considering if an artifact is reusable. Artifact evaluation should consider the methods of that evaluation alongside the varying of inputs to that evaluation. To achieve this, we suggest an extension to the scope of artifact badging, and describe both approaches and best practice arising in other communities. We seek to promote conversation and make a call to action intended to strengthen the scientific method within our domain.

中文翻译：

---

关于神器徽章的思考

可重复性：重复实验时获得一致结果的程度，作为验证实验结果、促进研究完整性和加速后续工作的重要手段。对工件审查和标记的承诺旨在提高可重复性并对提交工件的质量进行排名。但是，如本期所示，当前的标记方案，其重点是可重用工件，可能无法识别架构、实现或评估的限制。我们建议，为了提高对工件可重复性的洞察力，工件评估的深度和性质必须超越简单地考虑工件是否可重用。工件评估应考虑该评估的方法以及对该评估的输入的变化。为了达成这个，我们建议扩大工件标记的范围，并描述其他社区出现的方法和最佳实践。我们寻求促进对话并呼吁采取行动，以加强我们领域内的科学方法。