Skip to main content
SpringerLink
Log in

A Generative Adversarial Network to Denoise Depth Maps for Quality Improvement of DIBR-Synthesized Stereoscopic Images

  • Original Article
  • Published:
Journal of Electrical Engineering & Technology Aims and scope Submit manuscript

Abstract

Depth map quality is an important factor that affects the quality of synthesized stereoscopic images in stereoscopic visual communication systems using the depth image-based rendering (DIBR) technique. This paper proposes a method using a generative adversarial network (GAN) to denoise depth maps corrupted by several types of distortion. The generative network of the proposed GAN builds on convolutional layers, residual layers, and transposed convolutional layers with symmetric skip connections. The discriminative network of the proposed GAN is designed as a convolutional neural network. The generative network for denoising depth maps is trained with cropped depth maps where distortion is applied. Objective and subjective assessment of denoised depth maps and DIBR-synthesized stereoscopic images demonstrate that the proposed GAN effectively reduces the distortion in the depth maps and improves the quality of DIBR-synthesized stereoscopic images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al (2016) Tensorflow: a system for large-scale machine learning. In: 12th \(\{\)USENIX\(\}\) symposium on operating systems design and implementation (\(\{\)OSDI\(\}\) 16), pp 265–283

  2. Arjovsky M, Chintala S, Bottou L (2017) Wasserstein gan. arXiv preprint arXiv:1701.07875

  3. Fehn C (2004) Depth-image-based rendering (dibr), compression, and transmission for a new approach on 3d-tv. In: Stereoscopic displays and virtual reality systems XI, vol 5291, pp 93–104. International Society for Optics and Photonics

  4. Fleishman S, Drori I, Cohen-Or D (2004) Bilateral mesh denoising. In: ACM SIGGRAPH 2003 papers, pp 950–953 x

  5. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  6. Han L, Wu X, Liang W, Hou G, Jia Y (2010) Discriminative human action recognition in the learned hierarchical manifold space. Image Vis Comput 28(5):836–849

    Article  Google Scholar 

  7. Herrera D, Kannala J, Heikkilä J (2012) Joint depth and color camera calibration with distortion correction. IEEE Trans Pattern Anal Mach Intell 34(10):2058–2064

    Article  Google Scholar 

  8. Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in neural information processing systems, pp 6626–6637

  9. Hu W, Li X, Cheung G, Au O (2013) Depth map denoising using graph-based transform and group sparsity. In: 2013 IEEE 15th international workshop on multimedia signal processing (MMSP), pp 001–006. IEEE

  10. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1125–1134

  11. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980

  12. Kopf J, Cohen MF, Lischinski D, Uyttendaele M (2007) Joint bilateral upsampling. ACM Trans Graph ToG 26(3):96

    Article  Google Scholar 

  13. Kundu D, Choi LK, Bovik AC, Evans BL (2018) Perceptual quality evaluation of synthetic pictures distorted by compression and transmission. Sig Process Image Commun 61:54–72

    Article  Google Scholar 

  14. Kupyn O, Budzan V, Mykhailych M, Mishkin D, Matas J (2018) Deblurgan: blind motion deblurring using conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8183–8192

  15. Li F, Yu J, Chai J (2008) A hybrid camera for motion deblurring and depth map super-resolution. In: 2008 IEEE conference on computer vision and pattern recognition, pp 1–8. IEEE

  16. Liu A, Lin W, Narwaria M (2011) Image quality assessment based on gradient similarity. IEEE Trans Image Process 21(4):1500–1512

    MathSciNet  MATH  Google Scholar 

  17. Lu S, Ren X, Liu F (2014) Depth enhancement via low-rank matrix completion. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3390–3397

  18. Mao X, Li Q, Xie H, Lau RY, Wang Z, Paul Smolley S (2017) Least squares generative adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 2794–2802

  19. Nah S, Hyun Kim T, Mu Lee K (2017) Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3883–3891

  20. Nasr MAS, Al Rahmawy MF, Tolba A (2016) Multi-scale structural similarity index for motion. Structure 147:148

    Google Scholar 

  21. Pan J, Sun D, Pfister H, Yang MH (2016) Blind image deblurring using dark channel prior. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1628–1636

  22. Richardt C, Stoll C, Dodgson NA, Seidel HP, Theobalt C (2012) Coherent spatiotemporal filtering, upsampling and rendering of rgbz videos. In: Computer graphics forum, vol 31, pp 247–256. Wiley Online Library

  23. Roth S, Black MJ (2005) Fields of experts: a framework for learning image priors. In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), vol 2, pp 860–867. IEEE

  24. Scharstein D, Hirschmüller H, Kitajima Y, Krathwohl G, Nešić N, Wang X, Westling P (2014) High-resolution stereo datasets with subpixel-accurate ground truth. In: German conference on pattern recognition, pp 31–42. Springer

  25. Song R, Ko H, Kuo CCJ (2014) Mcl-3d: a database for stereoscopic image quality assessment using 2d-image-plus-depth source. J Inform Sci Eng 31

  26. Sterzentsenko V, Saroglou L, Chatzitofis A, Thermos S, Zioulis N, Doumanoglou A, Zarpalas D, Daras P (2019) Self-supervised deep depth denoising. In: Proceedings of the IEEE international conference on computer vision, pp 1242–1251

  27. Tanimoto M, Fujii T, Suzuki K (2009) View synthesis algorithm in view synthesis reference software 2.0 (vsrs2. 0). ISO/IEC JTC1/SC29/WG11 M 16090, 2009

  28. Wang KF, Gou C, Duan YJ, Lin YL, Zheng XH, Wang F (2017) Generative adversarial networks: the state of the art and beyond. Acta Automatica Sinica 43(3):321–332

    Google Scholar 

  29. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  30. Xie J, Feris RS, Yu SS, Sun MT (2015) Joint super resolution and denoising from a single depth image. IEEE Trans Multimed 17(9):1525–1537

    Article  Google Scholar 

  31. Xu L, Zheng S, Jia J (2013) Unnatural \(l_0\) sparse representation for natural image deblurring. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1107–1114

  32. Xue W, Zhang L, Mou X, Bovik AC (2013) Gradient magnitude similarity deviation: a highly efficient perceptual image quality index. IEEE Trans Image Process 23(2):684–695

    Article  MathSciNet  Google Scholar 

  33. Yoshizawa S, Belyaev A, Seidel HP (2006) Smoothing by example: mesh denoising by averaging with similarity-based weights. In: IEEE international conference on shape modeling and applications 2006 (SMI’06), p 9. IEEE

  34. Zaeemzadeh A, Rahnavard N, Shah M (2020) Norm-preservation: Why residual networks can become extremely deep? IEEE Trans Pattern Anal Mach Intell

  35. Zhang L, Jieyu Z, Xulun Y et al (2018) Co-operative generative adversarial nets. Acta Automatica Sinica 44(5):804–810

    MATH  Google Scholar 

  36. Zhang X, Wu R (2016) Fast depth image denoising and enhancement using a deep convolutional network. In: 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 2499–2503. IEEE

  37. Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 2223–2232

Download references

Author information

Authors and Affiliations

  1. School of Electronics and Information Engineering, Nanjing University of Information Science and Technology, Nanjing, 210044, Jiangsu, China

    Chuang Zhang, Xian-wen Sun, Xiao-yu Huang & Gui-yue Yu

  2. College of Computer Science and Artificial Intelligence, Wenzhou University, Wenzhou, China

    Jiawei Xu

  3. School of Software, Hallym University, Chunchon, Gangwon-do, Korea

    Seop Hyeong Park

Authors
  1. Chuang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xian-wen Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiawei Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiao-yu Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gui-yue Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Seop Hyeong Park
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Seop Hyeong Park.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This research was supported by Jiangsu Key Laboratory of Meteorological Observation and Information Processing Open Project (KDXS1805) and by the Priority Academic Program Development of Jiangsu Higher Education Institutions Project.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, C., Sun, Xw., Xu, J. et al. A Generative Adversarial Network to Denoise Depth Maps for Quality Improvement of DIBR-Synthesized Stereoscopic Images. J. Electr. Eng. Technol. 16, 2201–2210 (2021). https://doi.org/10.1007/s42835-021-00728-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42835-021-00728-2

Keywords

Search

Navigation