Abstract

In traditional semantic segmentation, knowing about all existing classes is essential to yield effective results with the majority of existing approaches. However, these methods trained in a Closed Set of classes fail when new classes are found in the test phase, not being able to recognize that an unseen class has been fed. This means that they are not suitable for Open Set scenarios, which are very common in real-world computer vision and remote sensing applications. In this paper, we discuss the limitations of Closed Set segmentation and propose two fully convolutional approaches to effectively address Open Set semantic segmentation: OpenFCN and OpenPCS. OpenFCN is based on the well-known OpenMax algorithm, configuring a new application of this approach in segmentation settings. OpenPCS is a fully novel approach based on feature-space from DNN activations that serve as features for computing PCA and multi-variate gaussian likelihood in a lower dimensional space. In addition to OpenPCS and aiming to reduce the RAM memory requirements of the methodology, we also propose a slight variation of the method (OpenIPCS) that uses an iteractive version of PCA able to be trained in small batches. Experiments were conducted on the well-known ISPRS Vaihingen/Potsdam and the 2018 IEEE GRSS Data Fusion Challenge datasets. OpenFCN showed little-to-no improvement when compared to the simpler and much more time efficient SoftMax thresholding, while being some orders of magnitude slower. OpenPCS achieved promising results in almost all experiments by overcoming both OpenFCN and SoftMax thresholding. OpenPCS is also a reasonable compromise between the runtime performances of the extremely fast SoftMax thresholding and the extremely slow OpenFCN, being able to run close to real-time. Experiments also indicate that OpenPCS is effective, robust and suitable for Open Set segmentation, being able to improve the recognition of unknown class pixels without reducing the accuracy on the known class pixels. We also tested the scenario of hiding multiple known classes to simulate multimodal unknowns, resulting in an even larger gap between OpenPCS/OpenIPCS and both SoftMax thresholding and OpenFCN, implying that gaussian modeling is more robust to settings with greater openness.

Graphic Abstract

中文翻译：

---

全卷积开放集分割

摘要

在传统的语义分割中，了解所有现有类别对于使用大多数现有方法产生有效结果至关重要。然而，当在测试阶段发现新类时，这些在类的封闭集合中训练的方法会失败，无法识别已经提供了一个看不见的类。这意味着它们不适合 Open Set 场景，后者在现实世界的计算机视觉和遥感应用中非常常见。在本文中，我们讨论了封闭集分割的局限性，并提出了两种完全卷积的方法来有效解决开放集语义分割：OpenFCN 和 OpenPCS。OpenFCN 基于著名的 OpenMax 算法，在分割设置中配置了这种方法的新应用。OpenPCS 是一种全新的方法，基于 DNN 激活的特征空间，用作计算低维空间中的 PCA 和多变量高斯似然的特征。除了 OpenPCS 并旨在减少该方法的 RAM 内存要求之外，我们还提出了该方法的轻微变体 (OpenIPCS)，该方法使用能够进行小批量训练的 PCA 迭代版本。实验是在著名的 ISPRS Vaihingen/Potsdam 和 2018 IEEE GRSS 数据融合挑战数据集上进行的。与更简单且时间效率更高的 SoftMax 阈值相比，OpenFCN 几乎没有改进，但速度慢了几个数量级。OpenPCS 通过克服 OpenFCN 和 SoftMax 阈值，在几乎所有实验中都取得了可喜的结果。OpenPCS 也是极快的 SoftMax 阈值和极慢的 OpenFCN 的运行时性能之间的合理折衷，能够接近实时地运行。实验还表明，OpenPCS 是有效的、鲁棒的并且适用于开放集分割，能够提高对未知类像素的识别，而不会降低对已知类像素的准确性。我们还测试了隐藏多个已知类以模拟多模态未知数的场景，导致 OpenPCS/OpenIPCS 与 SoftMax 阈值和 OpenFCN 之间的差距更大，这意味着高斯建模对具有更大开放性的设置更稳健。实验还表明，OpenPCS 是有效的、鲁棒的并且适用于开放集分割，能够提高对未知类像素的识别，而不会降低对已知类像素的准确性。我们还测试了隐藏多个已知类以模拟多模态未知数的场景，导致 OpenPCS/OpenIPCS 与 SoftMax 阈值和 OpenFCN 之间的差距更大，这意味着高斯建模对具有更大开放性的设置更稳健。实验还表明，OpenPCS 是有效的、鲁棒的并且适用于开放集分割，能够提高对未知类像素的识别，而不会降低对已知类像素的准确性。我们还测试了隐藏多个已知类以模拟多模态未知数的场景，导致 OpenPCS/OpenIPCS 与 SoftMax 阈值和 OpenFCN 之间的差距更大，这意味着高斯建模对具有更大开放性的设置更稳健。

图形摘要