Skip to main content
SpringerLink
Log in

Research on image inpainting algorithm of improved total variation minimization method

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

In order to solve the issue mismatching and structure disconnecting in exemplar-based image inpainting, an image completion algorithm based on improved total variation minimization method had been proposed in the paper, refer as ETVM. The structure of image had been extracted using improved total variation minimization method, and the known information of image is sufficiently used by existing methods. The robust filling mechanism can be achieved according to the direction of image structure and it has less noise than original image. The priority term had been redefined to eliminate the product effect and ensure data term had always effective. The priority of repairing patch and the best matching patch are determined by the similarity of the known information and the consistency of the unknown information in the repairing patch. The comparisons with cognitive computing image algorithms had been shown that the proposed method can ensure better selection of candidate image pixel to fill with, and it is achieved better global coherence of image completion than others. The inpainting results of noisy images show that the proposed method has good robustness and can also get good inpainting results for noisy images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Bengio YS, Courville A, Vincent P (2013) Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell 35(8):1798–1828

    Article  Google Scholar 

  • Bertalmio M, Sapiro G, Caselles V (2000) Image inpainting. In: Proceedings of Annual Conference on Computer Graphics and Interactive Techniques, New Orleans, LA, USA, 23–28 July 2000, pp 417–424

  • Cao F, Gousseau Y, Masnou S, Perez P (2011) Geometrically guided exemplar-based inpainting. SIAM J Appl Math 4(4):1143–1179

    MathSciNet  MATH  Google Scholar 

  • Chan TF, Shen JH (2001) Non-texture inpainting by curvature-driven diffusions (CDD). J Vis Commun Image Represent 4(12):436–449

    Article  Google Scholar 

  • Chan TF, Shen JH (2002) Mathematical models for local nonteture inpaintings. SIAM J Appl Math 62(3):1019–1043

    Article  MATH  Google Scholar 

  • Chen ZH, Dai CC, Liang L, Sheng B, Zhang J, Lin WY, Yuan YB (2016) Structure-aware image inpainting using patch scale optimization. J Vis Commun Image Represent 40(312):323

    Google Scholar 

  • Chen YT, Xiong J, Xu WH, Zuo JW (2019) A novel online incremental and decremental learning algorithm based on variable support vector machine. Cluster Comput 22(3): 7435–7445

    Article  Google Scholar 

  • Chen YT, Phonevilay V, Tao JJ, Chen X, Xia RL, Zhang Q, Yang K, Xiong J, Xie JB (2020a) The face image super-resolution algorithm based on combined representation learning. Multimed Tools Appl. https://doi.org/10.1007/s11042-020-09969-1

    Article  Google Scholar 

  • Chen YT, Tao JJ, Liu LW, Xiong J, Xia RL, Xie JB, Zhang Q, Yang K (2020b) Research of improving semantic image segmentation based on a feature fusion model. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-020-02066-z

    Article  Google Scholar 

  • Chen YT, Liu LW, Phonevilay V, Gu K, Xia RL, Xie JB, Zhang Q, Yang K (2020c) Image super-resolution reconstruction based on feature map attention mechanism. Appl Intell. https://doi.org/10.1007/s10489-020-02116-1

    Article  Google Scholar 

  • Chen YT, Liu LW, Tao JJ, Xia RL, Zhang Q, Yang K, Xiong J, Chen X (2020d) The improved image inpainting algorithm via encoder and similarity constraint. Visual Comput. https://doi.org/10.1007/s00371-020-01932-3

    Article  Google Scholar 

  • Chen YT, Liu LW, Tao JJ, Chen X, Xia RL, Zhang Q, Xiong J, Yang K, Xie JB (2020e) The image annotation algorithm using convolutional features from intermediate layer of deep learning. Multimed Tools Appl. https://doi.org/10.1007/s11042-020-09887-2

    Article  Google Scholar 

  • Chen YT, Zhang HP, Liu LW, Chen X, Zhang Q, Yang K, Xia RL, Xie JB (2020f) Research on image inpainting algorithm of improved GAN based on two-discriminations networks. Appl Intell. https://doi.org/10.1007/s10489-020-01971-2

    Article  Google Scholar 

  • Choi SY, Dowan C (2019) Unmanned aerial vehicles using machine learning for autonomous flight; state-of-the-art. Adv Robot 33:265–277

    Article  Google Scholar 

  • Criminisi A, Perez P, Toyama K (2004) Region filling and object removal by exemplar-based inpainting. IEEE Trans Image Process 13(9):1200–1212

    Article  Google Scholar 

  • Erfan BT, Ali ARH, Mehdi S, Kim JU (2018) Semi-supervised learning with generative adversarial networks on digital signal modulation classification. Comput Mater Cont 55:243–254

    Google Scholar 

  • Gao GW, Yu Y, Xie J, Yang J, Yang M, Zhang J (2020) Constructing multilayer locality-constrained matrix regression framework for noise robust face super-resolution. Pattern Recogn 107539. https://doi.org/10.1016/j.patcog.2020.107539

  • Ghorai M, Chanda B (2015) An image inpainting algorithm using PLSA-based search space estimation. Mach Vis Appl 26(1):69–87

    Article  Google Scholar 

  • Goodfellow I, Pouget-Adadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Proceedings of 2014 advances in neural information processing systems, Montreal, Quebec, Canada, 8–13 December 2014, pp 2672–2680

  • Gu K, Zhai G, Yang X, Zhang WJ (2015) Using free energy principle for blind image quality assessment. IEEE Trans Multimed 17:50–63

    Article  Google Scholar 

  • He KM, Sun J (2012) Computing nearest-neighbor fields via propagation assisted KD-trees. In: Proceedings of 2012 IEEE conference computer vision and pattern recognition, 16–21 June 2012, Providence, RI, pp 111–118

  • He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of 2016 IEEE conference on computer vision and pattern recognition, Las Vegas, USA, 27–30 June 2016, pp 770–778

  • Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Proceedings of 2012 International Conference neural information processing systems, Lake Tahoe, Nevada, 3–6 December 2012, pp 1097–1105

  • Kumar V, Mukherjee J, Mandal SKD (2016) Image inpainting through metric labelling via guided patch mixing. IEEE Trans Image Process 25(11):5212–5226

    Article  MathSciNet  MATH  Google Scholar 

  • Li Z, Wu J (2019) Learning deep CNN denoiser priors for depth image inpainting. Appl Sci 9:1103. https://doi.org/10.3390/app9061103

    Article  Google Scholar 

  • Li JY, Wang N, Zhang LF, Du B, Tao DC (2020) Recurrent feature reasoning for image inpainting. In: Proceedings of 2020 IEEE Conference on computer vision and pattern recognition, Seattle, WA, 13–19 June 2020, pp 7757–7765

  • Lu WP, Zhang YT, Wang SJ, Huang HY, Liu Q, Luo S (2020) Concept representation by learning explicit and implicit concept couplings. IEEE Intell Syst. https://doi.org/10.1109/MIS.2020.3021188

    Article  Google Scholar 

  • Min X, Ma K, Gu K, Zhai GT, Wang Z, Lin WS (2017) Unified blind quality assessment of compressed natural, graphic, and screen content images. IEEE Trans Image Process 26:5462–5474

    Article  MathSciNet  MATH  Google Scholar 

  • Nah S, Kim TH, Lee KM (2017) Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings of 2016 IEEE Conference on computer vision pattern recognition, Honolulu, HI, 21–26 July 2017, pp 257–265

  • Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros AA (2016) Context encoders: feature learning by inpainting. In: Proceedings of 2017 IEEE conference computer vision and pattern recognition, Las Vegas, NV, 27–30 June 2016, pp 2536–2544

  • Pierre B, Maxime D, David T, Olivier L (2015) Exemplar-based inpainting: technical review and new heuristics for better geometric reconstructions. IEEE Trans Image Process 24(6):1809–1823

    MathSciNet  MATH  Google Scholar 

  • Qiao TT, Zhang J, Xu DQ, Tao DC (2019) MirrorGAN: learning text-to-image generation by redescription. In: Proceedings of IEEE conference on computer vision and pattern recognition, Long Beach, CA, 16–20 June 2019. https://arxiv.org/pdf/1903.05854.pdf

  • Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR. https://arxiv.org/abs/1511.06434

  • Sun RX, Shi LF, Yin CY, Wang J (2019) An improved method in deep packet inspection based on regular expression. J Supercomput 75:3317–3333

    Article  Google Scholar 

  • Sun L, Wu FY, Zhan TM, Liu W, Wang J, Jeon B (2020) Weighted nonlocal low-rank tensor decomposition method for sparse unmixing of hyperspectral images. IEEE J Sel Top Appl Earth Obs Remote Sens. https://doi.org/10.1109/JSTARS.2020.2980576

    Article  Google Scholar 

  • Vese LA, Osher SJ (2004) Image denoising and decomposition with total variation minimization and oscillatory functions. J Math Imaging Vision 20(1–2):7–18

    Article  MathSciNet  MATH  Google Scholar 

  • Wang J, Ju CW, Gao Y, Lu HM, Yang WK, Gao H (2018) A PSO based energy efficient coverage control algorithm for wireless sensor networks. Comput Mater Cont 56:433–446

    Google Scholar 

  • Wexler Y, Shechtman E, Irani M (2007) Space-time completion of video. IEEE Trans Pattern Anal Mach Intell 29(3):463–476

    Article  Google Scholar 

  • Xu Z, Sun J (2010) Image inpainting by patch propagation using patch sparsity. IEEE Trans Image Process 19(5):1153–1165

    Article  MathSciNet  MATH  Google Scholar 

  • Yair N, Michaeli T (2018) Multi-scale weighted nuclear norm image restroation. In: Proceedings of 2018 IEEE conference on computer vision pattern recognition, Salt Lake City, UT, 16–22 June 2018, pp 3165–3174

  • Yeh AR, Chen C, Lim TY, Schwing AG (2017) Semantic image inpainting with deep generative models. In: Proceedings of 2017 IEEE conference computer vision and pattern recognition, Honolulu, HI, 26–26 July 2017, pp 6882–6890

  • Yi ZL, Tang Q, Azizi S, Jiang D, Xu Z (2020) Contextual residual aggregation for ultra high-resolution image inpainting. In: Proceedings of 2020 IEEE conference on computer vision and pattern recognition, Seattle, WA, 13–19 June 2020, pp 7505–7514

  • Yin YY, Chen L, Xu YS, Wan J (2018) Location-aware service recommendation with enhanced probabilistic matrix factorization. IEEE Access 6:62815–62825

    Article  Google Scholar 

  • Yin YY, Chen L, Xu YS, Wan J, Zhang H, Mai ZD (2019) QoS prediction for service recommendation with deep feature learning in edge computing environment. Mobile Netw Appl. https://doi.org/10.1007/s11036-019-01241-7

    Article  Google Scholar 

  • Yu Y, Tang SH, Aizawa K, Aizawa A (2019a) Category-based deep CCA for fine-grained venue discovery from multimodal data. IEEE Trans Neural Netw Learn Syst 30(4):1250–1258

    Article  MathSciNet  Google Scholar 

  • Yu JH, Lin Z, Yang JM, Shen XH, Lu X, Huang TS (2019b) Free-form image inpainting with gated convolution. In: Proceedings of 2019 IEEE/CVF International Conference on computer vision, Seoul, October 27–November 2, pp 4470–4479

  • Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: Proceedings of 2014 European Conference on computer vision, Zurich, 6–12 September 2014, pp 818–833

  • Zhang HM, Qian JJ, Gao JB, Yang J, Xu CY (2019) Scalable proximal jacobian iteration method with global convergence analysis for nonconvex unconstrained composite optimizations. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2018.2885699

    Article  MathSciNet  Google Scholar 

  • Zhang JM, Sun J, Wang J, Yue XG (2020a) Visual object tracking based on residual network and cascaded correlation filters. J Ambient Intell Humaniz Comp. https://doi.org/10.1007/s12652-020-02572-0

    Article  Google Scholar 

  • Zhang JM, Xie ZP, Sun J, Zou X, Wang J (2020b) A cascaded R-CNN with multiscale attention and imbalanced samples for traffic sign detection. IEEE Access 8:29742–29754

    Article  Google Scholar 

  • Zhou W, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

Download references

Acknowledgements

This research was funded by the National Natural Science Foundation of China [61972056, 61772454, 61402053], the Hunan Provincial Natural Science Foundation of China [2020JJ4623], the Scientific Research Fund of Hunan Provincial Education Department [17A007, 19C0028, 19B005], the Changsha Science and Technology Planning [KQ1703018, KQ1706064, KQ1703018-01, KQ1703018-04], the Junior Faculty Development Program Project of Changsha University of Science and Technology [2019QJCZ011], the “Double First-class” International Cooperation and Development Scientific Research Project of Changsha University of Science and Technology [2019IC34], the Practical Innovation and Entrepreneurship Ability Improvement Plan for Professional Degree Postgraduate of Changsha University of Science and Technology [SJCX202072], the Postgraduate Training Innovation Base Construction Project of Hunan Province [2019-248-51, 2020-172-48], the Beidou Micro Project of Hunan Provincial Education Department [XJT[2020] No.149].

Author information

Authors and Affiliations

  1. School of Computer and Communication Engineering and Hunan Provincial Key Laboratory of Intelligent Processing of Big Data On Transportation, Changsha University of Science and Technology, Changsha, 410114, Hunan, China

    Yuantao Chen, Haopeng Zhang, Linwu Liu & Jiajun Tao

  2. Department of Electronic Products, ZOOMLION Intelligent Technology Corporation Limited, Changsha, 410005, HunanHunan, China

    Qian Zhang & Kai Yang

  3. Hunan Institute of Scientific and Technical Information, Changsha, 410001, Hunan, China

    Runlong Xia & Jingbo Xie

Authors
  1. Yuantao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haopeng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Linwu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiajun Tao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qian Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kai Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Runlong Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jingbo Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuantao Chen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Y., Zhang, H., Liu, L. et al. Research on image inpainting algorithm of improved total variation minimization method. J Ambient Intell Human Comput 14, 5555–5564 (2023). https://doi.org/10.1007/s12652-020-02778-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-020-02778-2

Keywords

Search

Navigation