Skip to main content
Log in

Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

This paper focuses on the unsupervised domain adaptation of transferring the knowledge from the source domain to the target domain in the context of semantic segmentation. Existing approaches usually regard the pseudo label as the ground truth to fully exploit the unlabeled target-domain data. Yet the pseudo labels of the target-domain data are usually predicted by the model trained on the source domain. Thus, the generated labels inevitably contain the incorrect prediction due to the discrepancy between the training domain and the test domain, which could be transferred to the final adapted model and largely compromises the training process. To overcome the problem, this paper proposes to explicitly estimate the prediction uncertainty during training to rectify the pseudo label learning for unsupervised semantic segmentation adaptation. Given the input image, the model outputs the semantic segmentation prediction as well as the uncertainty of the prediction. Specifically, we model the uncertainty via the prediction variance and involve the uncertainty into the optimization objective. To verify the effectiveness of the proposed method, we evaluate the proposed method on two prevalent synthetic-to-real semantic segmentation benchmarks, i.e., GTA5 \(\rightarrow \) Cityscapes and SYNTHIA \(\rightarrow \) Cityscapes, as well as one cross-city benchmark, i.e., Cityscapes \(\rightarrow \) Oxford RobotCar. We demonstrate through extensive experiments that the proposed approach (1) dynamically sets different confidence thresholds according to the prediction variance, (2) rectifies the learning from noisy pseudo labels, and (3) achieves significant improvements over the conventional pseudo label learning and yields competitive performance on all three benchmarks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  • Blum, A., & Mitchell, T. (1998). Combining labeled and unlabeled data with co-training. In COLT.

  • Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2017). Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4), 834–848.

    Article  Google Scholar 

  • Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016). The cityscapes dataset for semantic urban scene understanding. In CVPR.

  • Fu, Y., Hospedales, T. M., Xiang, T., & Gong, S. (2015). Transductive multi-view zero-shot learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(11), 2332–2345.

    Article  Google Scholar 

  • Gal, Y., & Ghahramani, Z. (2016). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML.

  • Ganin, Y., & Lempitsky, V. (2015). Unsupervised domain adaptation by backpropagation. In ICML.

  • Grandvalet, Y., & Bengio, Y. (2005). Semi-supervised learning by entropy minimization. In NeurIPS.

  • Han, L., Zou, Y., Gao, R., Wang, L., & Metaxas, D. (2019). Unsupervised domain adaptation via calibrating uncertainties. In CVPR Workshops.

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016a). Deep residual learning for image recognition. In CVPR.

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016b). Identity mappings in deep residual networks. In ECCV.

  • Hendrycks, D., & Dietterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. In ICLR.

  • Hoffman, J., Tzeng, E., Park, T., Zhu, J.-Y., Isola, P., Saenko, K., Efros, A. A., & Darrell, T. (2018). Cycada: Cycle-consistent adversarial domain adaptation. In ICML.

  • Huang, H., Huang, Q., & Krahenbuhl, P. (2018). Domain transfer through deep activation matching. In ECCV.

  • Kang, G., Wei, Y., Yang, Y., & Hauptmann, A. (2020). Pixel-level cycle association: A new perspective for domain adaptive semantic segmentation. In NeurIPS.

  • Kendall, A., & Gal, Y. (2017). What uncertainties do we need in bayesian deep learning for computer vision? In NeurIPS.

  • Lee, D.-H. (2013). Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. ICML: In Workshop on challenges in representation learning.

  • Li, G., Kang, G., Liu, W., Wei, Y., & Yang, Y. (2020a). Content-consistent matching for domain adaptive semantic segmentation. In ECCV.

  • Li, P., Wei, Y., & Yang, Y. (2020). Consistent structural relation learning for zero-shot segmentationg. In NeurIPS.

  • Li, P., Wei, Y., & Yang, Y. (2020). Meta parsing networks: Towards generalized few-shot scene parsing with adaptive metric learning. In ACM Multimedia.

  • Liang, X., Lin, L., Wei, Y., Shen, X., Yang, J., & Yan, S. (2017). Proposal-free network for instance-level object segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12), 2978–2991.

    Article  Google Scholar 

  • Luo, Y., Liu, P., Guan, T., Yu, J., & Yang, Y. (2019a). Significance-aware information bottleneck for domain adaptive semantic segmentation. In ICCV.

  • Luo, Y., Liu, P., Guan, T., Yu, J., & Yang, Y. (2020). Adversarial style mining for one-shot unsupervised domain adaptation. In NeurIPS.

  • Luo, Y., Zheng, L., Guan, T., Yu, J., & Yang, Y. (2019b). Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. In CVPR.

  • Maddern, W., Pascoe, G., Linegar, C., & Newman, P. (2017). 1 Year, 1000 km: The Oxford RobotCar Dataset. The International Journal of Robotics Research (IJRR), 36(1), 3–15.

    Article  Google Scholar 

  • Nielsen, T. D., & Jensen, F. V. (2009). Bayesian networks and decision graphs. New York: Springer.

    MATH  Google Scholar 

  • Pan, Y., Yao, T., Li, Y., Wang, Y., Ngo, C.-W., & Mei, T. (2019). Transferrable prototypical networks for unsupervised domain adaptation. In CVPR.

  • Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., & Lerer, A. (2017). Automatic differentiation in pytorch.

  • Reed, S., Lee, H., Anguelov, D., Szegedy, C., Erhan, D., & Rabinovich, A. (2014). Training deep neural networks on noisy labels with bootstrapping. arXiv:1412.6596.

  • Richter, S. R., Vineet, V., Roth, S., & Koltun, V. (2016). Playing for data: Ground truth from computer games. In ECCV.

  • Ros, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A. M. (2016). The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR.

  • Saito, K., Watanabe, K., Ushiku, Y., & Harada, T. (2018). Maximum classifier discrepancy for unsupervised domain adaptation. In CVPR.

  • Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1), 1929–1958.

    MathSciNet  MATH  Google Scholar 

  • Tsai, Y.-H., Hung, W.-C., Schulter, S., Sohn, K., Yang, M.-H., & Chandraker, M. (2018). Learning to adapt structured output space for semantic segmentation. In CVPR.

  • Tsai, Y.-H., Sohn, K., Schulter, S., & Chandraker, M. (2019). Domain adaptation for structured output via discriminative representations. In ICCV.

  • Tzeng, E., Hoffman, J., Darrell, T., & Saenko, K. (2015). Simultaneous deep transfer across domains and tasks. In ICCV.

  • Vu, T.-H., Jain, H., Bucher, M., Cord, M., & Pérez, P. (2019). Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In CVPR.

  • Wang, J., Zhu, X., Gong, S., & Li, W. (2018). Transferable joint attribute-identity deep learning for unsupervised person re-identification. In CVPR.

  • Wang, S., Zhang, L., Zuo, W., & Zhang, B. (2019). Class-specific reconstruction transfer learning for visual recognition across domains. IEEE Transactions on Image Processing, 29, 2424–2438.

    Article  Google Scholar 

  • Wei, Y., Xiao, H., Shi, H., Jie, Z., Feng, J., & Huang, T. S. (2018). Revisiting dilated convolution: A simple approach for weakly-and semi-supervised semantic segmentation. In CVPR.

  • Wu, Z., Han, X., Lin, Y.-L., Gokhan Uzunbas, M., Goldstein, T., Nam Lim, S., & Davis, L. S. (2018). Dcan: Dual channel-wise alignment networks for unsupervised scene adaptation. In ECCV.

  • Wu, Z., Wang, X., Gonzalez, J. E., Goldstein, T., & Davis, L. S. (2019). Ace: adapting to changing environments for semantic segmentation. In ICCV.

  • Yang, J., Xu, R., Li, R., Qi, X., Shen, X., Li, G., & Lin, L. (2020). An adversarial perturbation oriented domain adaptation approach for semantic segmentation. In AAAI.

  • Yu, T., Li, D., Yang, Y., Hospedales, T. M., & Xiang, T. (2019). Robust person re-identification by modelling feature uncertainty. In ICCV.

  • Yue, X., Zhang, Y., Zhao, S., Sangiovanni-Vincentelli, A., Keutzer, K., & Gong, B. (2019). Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data. In ICCV.

  • Zhang, L., Li, X., Arnab, A., Yang, K., Tong, Y., & Torr, P. H. (2019a). Dual graph convolutional network for semantic segmentation. In BMVC.

  • Zhang, L., Wang, S., Huang, G.-B., Zuo, W., Yang, J., & Zhang, D. (2019b). Manifold criterion guided transfer learning via intermediate domain generation. IEEE Transactions on Neural Networks and Learning Systems, 30(12), 3759–3773.

    Article  MathSciNet  Google Scholar 

  • Zhang, L., Xu, D., Arnab, A., and Torr, P. H. (2020). Dynamic graph message passing network. In CVPR.

  • Zhang, Y., Qiu, Z., Yao, T., Liu, D., & Mei, T. (2018). Fully convolutional adaptation networks for semantic segmentation. In CVPR.

  • Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017). Pyramid scene parsing network. In CVPR.

  • Zheng, Z., & Yang, Y. (2020). Unsupervised scene adaptation with memory regularization in vivo. In IJCAI.

  • Zou, Y., Yu, Z., Liu, X., Kumar, B., & Wang, J. (2019). Confidence regularized self-training. In ICCV.

  • Zou, Y., Yu, Z., Vijaya Kumar, B., & Wang, J. (2018). Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In ECCV.

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yi Yang.

Additional information

Communicated by Andreas Geiger.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zheng, Z., Yang, Y. Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation. Int J Comput Vis 129, 1106–1120 (2021). https://doi.org/10.1007/s11263-020-01395-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-020-01395-y

Keywords

Navigation