Detecting Out-of-Distribution (OoD) data is one of the greatest challenges in safe and robust deployment of machine learning algorithms in medicine. When the algorithms encounter cases that deviate from the distribution of the training data, they often produce incorrect and over-confident predictions. OoD detection algorithms aim to catch erroneous predictions in advance by analysing the data distribution and detecting potential instances of failure. Moreover, flagging OoD cases may support human readers in identifying incidental findings. Due to the increased interest in OoD algorithms, benchmarks for different domains have recently been established. In the medical imaging domain, for which reliable predictions are often essential, an open benchmark has been missing. We introduce the Medical-Out-Of-Distribution-Analysis-Challenge (MOOD) as an open, fair, and unbiased benchmark for OoD methods in the medical imaging domain. The analysis of the submitted algorithms shows that performance has a strong positive correlation with the perceived difficulty, and that all algorithms show a high variance for different anomalies, making it yet hard to recommend them for clinical practice. We also see a strong correlation between challenge ranking and performance on a simple toy test set, indicating that this might be a valuable addition as a proxy dataset during anomaly detection algorithm development.

中文翻译：

---

MOOD 2020：医学图像分布外检测和定位的公共基准

检测分布外 (OoD) 数据是在医学中安全、稳健地部署机器学习算法的最大挑战之一。当算法遇到偏离训练数据分布的情况时，它们通常会产生不正确和过度自信的预测。OoD 检测算法旨在通过分析数据分布和检测潜在的故障实例来提前捕捉错误的预测。此外，标记 OoD 案例可能有助于人类读者识别偶然发现。由于对 OoD 算法的兴趣增加，最近建立了不同领域的基准。在医学成像领域，可靠的预测通常是必不可少的，因此缺少一个开放的基准。我们引入了医学分布分析挑战 (MOOD) 作为医学成像领域中 OoD 方法的开放、公平和无偏见的基准。对提交的算法的分析表明，性能与感知难度有很强的正相关，并且所有算法对于不同的异常都表现出很大的差异，因此很难推荐它们用于临床实践。我们还在一个简单的玩具测试集上看到了挑战排名和性能之间的强相关性，这表明这可能是在异常检测算法开发过程中作为代理数据集的有价值的补充。并且所有算法对于不同的异常都显示出很大的差异，因此很难将它们推荐用于临床实践。我们还在一个简单的玩具测试集上看到了挑战排名和性能之间的强相关性，这表明这可能是在异常检测算法开发过程中作为代理数据集的有价值的补充。并且所有算法对于不同的异常都显示出很大的差异，因此很难将它们推荐用于临床实践。我们还在一个简单的玩具测试集上看到了挑战排名和性能之间的强相关性，这表明这可能是在异常检测算法开发过程中作为代理数据集的有价值的补充。