Deep learning has enabled impressive progress in the accuracy of semantic segmentation. Yet, the ability to estimate uncertainty and detect failure is key for safety-critical applications like autonomous driving. Existing uncertainty estimates have mostly been evaluated on simple tasks, and it is unclear whether these methods generalize to more complex scenarios. We present Fishyscapes, the first public benchmark for anomaly detection in a real-world task of semantic segmentation for urban driving. It evaluates pixel-wise uncertainty estimates towards the detection of anomalous objects. We adapt state-of-the-art methods to recent semantic segmentation models and compare uncertainty estimation approaches based on softmax confidence, Bayesian learning, density estimation, image resynthesis, as well as supervised anomaly detection methods. Our results show that anomaly detection is far from solved even for ordinary situations, while our benchmark allows measuring advancements beyond the state-of-the-art. Results, data and submission information can be found at https://fishyscapes.com/.

深度学习在语义分割的准确性方面取得了令人瞩目的进步。然而，估计不确定性和检测故障的能力是自动驾驶等安全关键应用的关键。现有的不确定性估计大多是在简单任务上进行评估的，尚不清楚这些方法是否可以推广到更复杂的场景。我们展示了 Fishyscapes，这是在城市驾驶语义分割的现实世界任务中进行异常检测的第一个公共基准。它评估异常对象检测的像素级不确定性估计。我们将最先进的方法应用于最近的语义分割模型，并比较基于 softmax 置信度、贝叶斯学习、密度估计、图像再合成以及监督异常检测方法的不确定性估计方法。我们的结果表明，即使在普通情况下，异常检测也远未解决，而我们的基准测试允许衡量超越最先进技术的进步。结果、数据和提交信息可在 https://fishyscapes.com/ 上找到。