Skip to main content
SpringerLink
Log in

Iterative feature refinement with network-driven prior for image restoration

  • Theoretical Advances
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

Image restoration (IR) has been extensively studied with lots of excellent strategies accumulated over the years. However, most existing methods still have large room for improvement. In this paper, we boost an unsupervised iterative feature refinement model (IFR) with the enhanced high-dimensional deep mean-shift prior (EDMSP), termed IFR-EDMSP. The proposed model inherits the fantastic noise suppression characteristic of embedded network and the fine detail preservation ability of IFR model. Moreover, based on the fact that multiple implementations of artificial noise in prior learning improve underlying representation capability, three-sigma rule is adopted in IFR-EDMSP for accurate and robust results. Extensive experiments demonstrated that IFR-EDMSP outperforms the typical methods in compressed sensing, image deblurring and super-resolution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Engl HW, Kunisch K, Neubauer A (1989) Convergence rates for Tikhonov regularisation of non-linear ill-posed problems[J]. Inverse Prob 5(4):523–540

    Article  Google Scholar 

  2. Rudin LI, Osher S, Fatemi E (1992) Nonlinear total variation based noise removal algorithms[J]. Physica D 60(1–4):259–268

    Article  MathSciNet  Google Scholar 

  3. Chambolle A (2004) An algorithm for total variation minimization and applications[J]. J Math imaging vision 20(1–2):89–97

    MathSciNet  MATH  Google Scholar 

  4. Elad M, Aharon M (2006) Image denoising via sparse and redundant representations over learned dictionaries[J]. IEEE Trans Image Process 15(12):3736–3745

    Article  MathSciNet  Google Scholar 

  5. Yang J, Wright J, Huang T, et al (2008) Image super-resolution as sparse representation of raw image patches[C]. IEEE conference on computer vision and pattern recognition. IEEE, pp 1–8.

  6. Mairal J, Elad M, Sapiro G (2007) Sparse representation for color image restoration[J]. IEEE Trans Image Process 17(1):53–69

    Article  MathSciNet  Google Scholar 

  7. Dong W, Li X, Zhang L, et al (2011) Sparsity-based image denoising via dictionary learning and structural clustering[C]. CVPR 2011. IEEE, pp 457–464.

  8. Qu X, Guo D, Ning B et al (2012) Undersampled MRI reconstruction with patch-based directional wavelets[J]. Magn Reson Imaging 30(7):964–977

    Article  Google Scholar 

  9. Liu Q, Wang S, Ying L et al (2013) Adaptive dictionary learning in sparse gradient domain for image recovery[J]. IEEE Trans Image Process 22(12):4652–4663

    Article  MathSciNet  Google Scholar 

  10. Liu Q, Wang S, Yang K et al (2013) Highly undersampled magnetic resonance image reconstruction using two-level Bregman method with dictionary updating[J]. IEEE Trans Med Imaging 32(7):1290–1301

    Article  Google Scholar 

  11. Zhu Y, Zhang Q, Liu Q et al (2015) PANDA-T1ρ: Integrating principal component analysis and dictionary learning for fast T 1 ρ mapping[J]. Magn Reson Med 73(1):263–272

    Article  Google Scholar 

  12. Buades A, Coll B, Morel J M (2005) A non-local algorithm for image denoising[C].2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05). IEEE, 2: 60–65.

  13. Dabov K, Foi A, Katkovnik V et al (2007) Image denoising by sparse 3-D transform-domain collaborative filtering[J]. IEEE Trans Image Process 16(8):2080–2095

    Article  MathSciNet  Google Scholar 

  14. Liang D, Wang H, Chang Y et al (2011) Sensitivity encoding reconstruction with nonlocal total variation regularization[J]. Magn Reson Med 65(5):1384–1392

    Article  Google Scholar 

  15. Zoran D, Weiss Y (2011) From learning models of natural image patches to whole image restoration[C]. 2011 International Conference on Computer Vision. IEEE, pp 479–486.

  16. Dong W, Zhang L, Shi G et al (2012) Nonlocally centralized sparse representation for image restoration[J]. IEEE Trans Image Process 22(4):1620–1630

    Article  MathSciNet  Google Scholar 

  17. Dong W, Shi G, Li X et al (2014) Compressive sensing via nonlocal low-rank regularization[J]. IEEE Trans Image Process 23(8):3618–3632

    Article  MathSciNet  Google Scholar 

  18. Gu S, Zhang L, Zuo W, et al (2014) Weighted nuclear norm minimization with application to image denoising[C]. Proceedings of the IEEE conference on computer vision and pattern recognition. pp 2862–2869.

  19. Eksioglu EM (2016) Decoupled algorithm for MRI reconstruction using nonlocal block matching model: BM3D-MRI[J]. J Math Imaging Vision 56(3):430–440

    Article  MathSciNet  Google Scholar 

  20. Levin A, Fergus R, Durand F, et al (2007) Image and depth from a conventional camera with a coded aperture[J]. ACM transactions on graphics (TOG), 26(3): 70-es.

  21. Bregman LM (1967) The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming[J]. USSR Comput Math Math Phys 7(3):200–217

    Article  MathSciNet  Google Scholar 

  22. Huang F, Chen Y, Yin W et al (2010) A rapid and robust numerical algorithm for sensitivity encoding with sparsity constraints: Self-feeding sparse SENSE[J]. Magn Reson Med 64(4):1078–1088

    Article  Google Scholar 

  23. Osher S, Burger M, Goldfarb D et al (2005) An iterative regularization method for total variation-based image restoration[J]. Multiscale Model Simul 4(2):460–489

    Article  MathSciNet  Google Scholar 

  24. Wang S, Liu J, Liu Q et al (2016) Iterative feature refinement for accurate undersampled MR image reconstruction[J]. Phys Med Biol 61(9):3291

    Article  Google Scholar 

  25. Cheng J, Jia S, Ying L et al (2018) Improved parallel image reconstruction using feature refinement[J]. Magn Reson Med 80(1):211–223

    Article  Google Scholar 

  26. Burger H C, Schuler C J, Harmeling (2012) Image denoising: Can plain neural networks compete with BM3D?[C]. 2012 IEEE conference on computer vision and pattern recognition. IEEE, pp 2392–2399.

  27. Chen E (2012) Image denoising and inpainting with deep neural networks[C]. Advances in neural information processing systems. pp 341–349.

  28. Divakar N, Venkatesh Babu R (2017) Image denoising via CNNs: an adversarial approach[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp 80–87.

  29. Dong C, Loy C C, He K, et al (2014) Learning a deep convolutional network for image super-resolution[C]. European conference on computer vision. Springer, Cham, pp 184–199.

  30. Wang Z, Liu D, Yang J, et al (2015) Deep networks for image super-resolution with sparse prior[C]. Proceedings of the IEEE international conference on computer vision. pp 370–378.

  31. Dong C, Loy C C, Tang X (2016) Accelerating the super-resolution convolutional neural network[C]. European conference on computer vision. Springer, Cham, pp 391–407.

  32. Kim J, Kwon Lee J, Mu Lee K (2016) Accurate image super-resolution using very deep convolutional networks[C]. Proceedings of the IEEE conference on computer vision and pattern recognition. pp 1646–1654.

  33. Dong C, Loy CC, He K et al (2015) Image super-resolution using deep convolutional networks[J]. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  Google Scholar 

  34. Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network[C]. Proceedings of the IEEE conference on computer vision and pattern recognition. pp 3147–3155.

  35. Zhang K, Zuo W, Gu S, et al (2017) Learning deep CNN denoiser prior for image restoration[C]. Proceedings of the IEEE conference on computer vision and pattern recognition. pp 3929–3938.

  36. Chen Y, Pock T (2016) Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration[J]. IEEE Trans Pattern Anal Mach Intell 39(6):1256–1272

    Article  Google Scholar 

  37. Tong T, Li G, Liu X, et al (2017) Image super-resolution using dense skip connections[C]. Proceedings of the IEEE International Conference on Computer Vision. pp 4799–4807.

  38. Wang S, Su Z, Ying L, et al (2016) Accelerating magnetic resonance imaging via deep learning[C]. 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI). IEEE, pp 514–517.

  39. Sun J, Li H, Xu Z (2016) Deep ADMM-Net for compressive sensing MRI[C]. Advances in neural information processing systems. pp 10–18.

  40. Zhang J, Ghanem B (2018) ISTA-Net: Interpretable optimization-inspired deep network for image compressive sensing[C]. Proceedings of the IEEE conference on computer vision and pattern recognition. pp1828–1837.

  41. Zhang Y, Sun L, Yan C et al (2018) Adaptive residual networks for high-quality image restoration[J]. IEEE Trans Image Process 27(7):3150–3163

    Article  MathSciNet  Google Scholar 

  42. Liu P, Zhang H, Zhang K, et al (2018) Multi-level wavelet-CNN for image restoration[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp 773–782.

  43. Zhang K, Zuo W, Chen Y et al (2017) Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising[J]. IEEE Trans Image Process 26(7):3142–3155

    Article  MathSciNet  Google Scholar 

  44. S. A. Bigdeli and M. Zwicker (2018) Image restoration using auto encoding priors[C]. In Proceedings of the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 5: 33–44.

  45. Wang Y, Liu Q, Zhou H et al (2018) Learning multi-denoising autoencoding priors for image super-resolution[J]. J Vis Commun Image Represent 57:152–162

    Article  Google Scholar 

  46. Bigdeli S A, Zwicker M, Favaro P, et al (2017) Deep mean-shift priors for image restoration[C]. Advances in Neural Information Processing Systems. pp 763–772.

  47. Liu R, Ma L, Wang Y et al (2018) Learning converged propagations with deep prior ensemble for image enhancement[J]. IEEE Trans Image Process 28(3):1528–1543

    Article  MathSciNet  Google Scholar 

  48. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation[C]. International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015: 234–241.

  49. Agustsson E, Timofte R (2017) Ntire 2017 challenge on single image super-resolution: Dataset and study[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp 126–135.

  50. Liu Q, Leung H (2019) Variable augmented neural network for decolorization and multi-exposure fusion[J]. Information Fusion 46:114–127

    Article  Google Scholar 

  51. Zhu M, Li J, Wang N, et al. Knowledge Distillation for Face Photo-Sketch Synthesis[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020.

  52. Zhu M, Li J, Wang N et al (2019) A deep collaborative framework for face photo–sketch synthesis[J]. IEEE transactions on neural networks and learning systems 30(10):3096–3108

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under 61871206, 61661031. and project of innovative special funds for graduate students YC2019-S052.

Author information

Authors and Affiliations

  1. Nanchang University, Nanchang, Jiangxi Province, 330031, P. R. China

    Jinjie Zhou, Miaomiao Meng, Jinglong Xing, Xiaoling Xu & Yinghong Zhang

  2. University of Florida, Gainesville, FL, 32611, USA

    Yuchen Xiong

Authors
  1. Jinjie Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Miaomiao Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jinglong Xing
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuchen Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaoling Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yinghong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yinghong Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, J., Meng, M., Xing, J. et al. Iterative feature refinement with network-driven prior for image restoration. Pattern Anal Applic 24, 1623–1634 (2021). https://doi.org/10.1007/s10044-021-01006-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-021-01006-7

Keywords

Search

Navigation