Pointly-supervised scene parsing with uncertainty mixture

Highlights

• We identify the existence of a statistical phenomenon called uncertainty mixture.

• We propose a threshold-free method to harvest pseudo labels.

• We contribute a novel regularized Gamma mixture model.

• We achieve state-of-the-art results on PascalContext and ADE20k.

Abstract

Pointly-supervised learning is an important topic for scene parsing, as dense annotation is extremely expensive and hard to scale. The state-of-the-art method harvests pseudo labels by applying thresholds upon softmax outputs (logits). There are two issues with this practice: (1) Softmax output does not necessarily reflect the confidence of the network output. (2) There is no principled way to decide on the optimal threshold. Tuning thresholds can be time-consuming for deep neural networks. Our method, by contrast, builds upon uncertainty measures instead of logits and is free of threshold tuning. We motivate the method with a large-scale analysis of the distribution of uncertainty measures, using strong models and challenging databases. This analysis leads to the discovery of a statistical phenomenon called uncertainty mixture. Specifically speaking, for each independent category, the distribution of uncertainty measures for unlabeled points is a mixture of two components (certain v.s. uncertain samples). The phenomenon of uncertainty mixture is surprisingly ubiquitous in real-world datasets like PascalContext and ADE20k. Inspired by this discovery, we propose to decompose the distribution of uncertainty measures with a Gamma mixture model, leading to a principled method to harvest reliable pseudo labels. Beyond that, we assume the uncertainty measures for labeled points are always drawn from the certain component. This amounts to a regularized Gamma mixture model. We provide a thorough theoretical analysis of this model, showing that it can be solved with an EM-style algorithm with convergence guarantee. Our method is also empirically successful. On PascalContext and ADE20k, we achieve clear margins over the baseline, notably with no threshold tuning in the pseudo label generation procedure. On the absolute scale, since our method collaborates well with strong baselines, we reach new state-of-the-art performance on both datasets.

Introduction

Dense annotation for scene parsing is very expensive. According to Mottaghi et al. (2014), it took the authors 3 to 5 min to annotate one PASCAL (Everingham et al., 2010) image with dense scene segments. Caesar et al. (2018) exploit superpixels to speed up the annotation of COCO (Lin et al., 2014) images, while still using 3 min on average. Thus annotating scenes with point clicks and semantic class assignment is an appealing alternative, and training with this kind of labels is called pointly-supervised scene parsing in this paper.¹ Specifically speaking, full supervision corresponds to the right-bottom parts of each panel in Fig. 1, and point supervision is enlarged and overlapped onto the input images as the left-top parts demonstrate. This setting is quite challenging because more than 99% regions in the images are not annotated. Besides, training with few samples is an exciting topic, which may help us understand the nature of modern models. For example, we identify a previously unknown statistical phenomenon called uncertainty mixture, which is an intriguing property of deep neural networks. In summary of this paragraph, the problem of pointly-supervised scene parsing is (1) urgent, (2) difficult and (3) potentially fruitful.

This work focuses on harvesting pseudo labels for the pointly-supervised training set. Specifically speaking, one can train a model using only the point supervision, which is referred to as the first round model. This model would produce a semantic label prediction for every pixel in the training set, as demonstrated in the right-top parts of each panel in Fig. 1. They are referred to as pseudo labels throughout this paper. They are erroneous, but containing right ones. The central problem considered here is how to harvest as-many-as-possible good pseudo labels for training. The state-of-the-art method (Qian et al., 2019) expands point supervision into regions and regards them as trustworthy pseudo labels. It is proven effective on public benchmarks yet there exist two critical issues:

Firstly, it relies upon pixel-wise softmax outputs, or say logits. It is known that logits can be over-confident upon wrong prediction (Li and Hoiem, 2018). The reason behind is that softmax result is only a single point estimate of the predictive distribution. Instead, we compute the uncertainty measures for these predictions, depicted in the left-bottom parts of each panel in Fig. 1. They faithfully reflect the confidence of the network outputs. It is obvious that harvesting pseudo labels with low uncertainty (low color temperature in Fig. 1) is a promising solution. However, how do we properly define ’low uncertainty’?

Secondly, harvesting pseudo labels using logits would introduce thresholds. It is very time-consuming to tune thresholds for modern deep networks. Meanwhile, using the natural thresholds generated by argmax would lead to a trivial usage of the pseudo labels. One may argue that using uncertainty measures still involves defining a low uncertainty threshold, as just mentioned in the last bullet point. We show that this problem can be resolved by exploiting a newly discovered statistical phenomenon called uncertainty mixture. It allows us to decide on the optimal threshold of uncertainty measures in an automatic way. Specifically speaking, we decompose the uncertainty measures for unlabeled points with a Gamma mixture and harvest pseudo labels belonging to the certain component.

Beyond the direct application of Gamma mixture, a new regularized model tailored for our problem is proposed, analyzed, implemented and evaluated. We assume the uncertainty measures for labeled points are drawn from the certain component. Intuitively, this helps the mixture model to better capture the shape of the certain component. Mathematically, this amounts to an added regularization term in the objective function of an EM procedure. Since the model has not been visited before, we present a systematic exposition of its analytical properties, from convergence guarantee to solver details. Empirically, this regularized Gamma mixture harvests pseudo labels of higher quality than the baseline and leads to better scene parsing performance, in all experimental settings we inspected.

Last but not least, our method is extensively benchmarked on challenging public datasets, namely PascalContext and ADE20k. It turns out that our method works robustly in various settings. This robustness is attributed to the nature of the method: our mixture modeling decides on the optimal threshold of uncertainty measures automatically. On an absolute scale, our method collaborates well with strong network architectures and training techniques, resulting in new state-of-the-art performance on both datasets. We believe our solution to be a useful one due to its robustness and good performance.

In summary, we claim four contributions for this study, in a descending order of generalization and importance:

• While uncertainty in deep learning is drawing more attention recently, understanding towards its behaviors in large-scale problems is still sparse. We present an analysis in the field of pointly-supervised scene parsing. The existence of a previously unknown statistical phenomenon called uncertainty mixture is revealed. By existence, we mean: (1) it is ubiquitously observed in real-world datasets; (2) it is understandable considering the nature of uncertainty; (3) the empirical success of our method (inspired by the phenomenon) retrospectively supports its existence.

• Encouraged by the existence of uncertainty mixture, a newmethodology is proposed: to decompose the distribution of uncertainty measures into certain and uncertain components and harvest pseudo labels belonging to the certain component. Under this perspective, those two critical issues mentioned above are resolved in a principled manner.

• A novel mixture model is introduced, which is tightly related to a unique feature of the problem considered. Making full use of labeled points, the uncertainty measures for them are formulated as a regularization term in the objective function. A detailed analytical exposition is provided, covering important aspects of this new model.

• Extensive evaluations on public benchmarks are conducted and analyzed. With ablative studies, we validate the effectiveness of key ingredients in our methods. On an absolute scale, we achieve new state-of-the-art results on PascalContext and ADE20k. This method won the pointly-supervised scene parsing track of CVPR 2020 challenge on learning from imperfect data.