Skip to main content
SpringerLink
Log in

A novel entropy-based texture inpainting algorithm

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Image inpainting is the process of restoring a lost or damaged portion of an image. Inpainting of an image that contains texture remains a particularly challenging problem. We aim to propose an algorithm to inpaint a textured image accurately using a single image. The main idea is to segment the given image, based on its texture. In this work, we propose a novel local energy approach, in combination with the k-means algorithm to segment the given image, based on its texture. We use this segmentation result to restrict the search of matching pixels to only-relevant segments. Moreover, we use the entropy-based dissimilarity parameter to find matching pixels, instead of the \(\ell ^2\) distance. The restriction of the search area improves the efficiency, and the use of the proposed dissimilarity parameter provides a better way to compare textures, giving improved inpainting for textured images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. Since the local entropy is rotation-invariant, \({\mathscr {D}}_{_{(x_0, y_0)}}(x,y)=0\) does not imply that \(x=y\), and hence \({\mathscr {D}}_{_{(x_0, y_0)}}(\cdot , \cdot )\) is not a metric.

  2. The experiments were done with MATLAB R2018b on 2.9 GHz Intel Core i9 processor.

References

  1. Aubert, G., Kornprobst, P.: Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations, vol. 147. Springer, Berlin (2006)

    Book  Google Scholar 

  2. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: Patchmatch: A randomized correspondence algorithm for structural image editing. In: ACM Transactions on Graphics (ToG), vol. 28, p. 24. ACM (2009)

  3. Bertalmio, M., Bertozzi, A., Sapiro, G.: Navier-stokes, fluid dynamics, and image and video inpainting. In: Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, vol. 1, pp. I–I. IEEE (2001)

  4. Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 417–424. ACM Press/Addison-Wesley Publishing Co. (2000)

  5. Bertalmio, M., Vese, L., Sapiro, G., Osher, S.: Simultaneous structure and texture image inpainting. IEEE Trans. Image Process. 12(8), 882–889 (2003)

    Article  Google Scholar 

  6. Chan, T.: Local inpainting models and TV inpainting. SIAM J. Appl. Math. 62(3), 1019–1043 (2001)

    MathSciNet  Google Scholar 

  7. Chan, T., Shen, J.: Non-texture inpainting by curvature-driven diffusions (CDD). J. Vis. Commun. Image Represent. 12(4), 436–449 (2001)

    Article  Google Scholar 

  8. Chan, T., Shen, J.: Image Processing and Analysis: Variational, PDE, Wavelet, and Stochastic Methods, vol. 94. SIAM, Philadelphia (2005)

    Book  Google Scholar 

  9. Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)

    Article  Google Scholar 

  10. Duan, J., Pan, Z., Zhang, B., Liu, W., Tai, X.C.: Fast algorithm for color texture image inpainting using the non-local CTV model. J. Glob. Optim. 62(4), 853–876 (2015)

    Article  MathSciNet  Google Scholar 

  11. Duval, V., Aujol, J.F., Vese, L.A.: Mathematical modeling of textures: application to color image decomposition with a projected gradient algorithm. J. Math. Imaging Vis. 37(3), 232–248 (2010)

    Article  MathSciNet  Google Scholar 

  12. Elad, M., Starck, J.L., Querre, P., Donoho, D.L.: Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA). Appl. Comput. Harmonic Anal. 19(3), 340–358 (2005)

    Article  MathSciNet  Google Scholar 

  13. Gonzalez, R.C., Woods, R.E.: Digital Image Processing. Pearson Publishing, London (2018)

    Google Scholar 

  14. He, K., Sun, J.: Statistics of patch offsets for image completion. In: European Conference on Computer Vision, pp. 16–29. Springer (2012)

  15. Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=Hk99zCeAb

  16. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: active contour models. Int. J. Comput. Vis. 1(4), 321–331 (1988)

    Article  Google Scholar 

  17. Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 85–100 (2018)

  18. Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: The European Conference on Computer Vision (ECCV) (2018)

  19. Lloyd, S.P.: Least squares quantization in PCM. IEEE Trans. Inf. Theory 28(2), 129–137 (1982)

    Article  MathSciNet  Google Scholar 

  20. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536–2544 (2016)

  21. Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. CoRR arXiv:1604.07379 (2016)

  22. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  23. Schönlieb, C.B.: Partial Differential Equation Methods for Image Inpainting, vol. 29. Cambridge University Press, Cambridge (2015)

    Book  Google Scholar 

  24. Shibata, T., Iketani, A., Senda, S.: Fast and structure-preserving inpainting based on probabilistic structure estimation. In: MVA, pp. 22–25 (2011)

  25. Starck, J.L., Elad, M., Donoho, D.L.: Image decomposition via the combination of sparse representations and a variational approach. IEEE Trans. Image Process. 14(10), 1570–1582 (2005)

    Article  MathSciNet  Google Scholar 

  26. Szeliski, R.: Computer Vision: Algorithms and Applications. Springer, Berlin (2010)

    MATH  Google Scholar 

  27. Thorndike, R.L.: Who belongs in the family? Psychometrika 18(4), 267–276 (1953)

    Article  Google Scholar 

  28. Wang, Y., Tao, X., Qi, X., Shen, X., Jia, J.: Image inpainting via generative multi-column convolutional neural networks. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, pp. 329–338. Curran Associates Inc., Red Hook, NY, USA (2018)

  29. Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using multi-scale neural patch synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6721–6729 (2017)

  30. Yeh, R.A., Chen, C., Yian Lim, T., Schwing, A.G., Hasegawa-Johnson, M., Do, M.N.: Semantic image inpainting with deep generative models. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

  31. Ying, H., Kai, L., Ming, Y.: An improved image inpainting algorithm based on image segmentation. Proc. Comput. Sci. 107, 796–801 (2017)

    Article  Google Scholar 

  32. Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: a 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 40(6), 1452–1464 (2018)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, Clarkson University, Potsdam, NY, 13669, USA

    Prashant Athavale

  2. Department of Computer Science, Clarkson University, Potsdam, NY, 13669, USA

    Soumyabrata Dey

  3. Department of Mathematics, Indian Institute of Science Education and Research, Thiruvananthapuram, India

    Sheetal Dharmatti & Aiswarya Sara Mathew

Authors
  1. Prashant Athavale
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Soumyabrata Dey
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sheetal Dharmatti
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Aiswarya Sara Mathew
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Prashant Athavale.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 2622 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Athavale, P., Dey, S., Dharmatti, S. et al. A novel entropy-based texture inpainting algorithm. SIViP 15, 1075–1080 (2021). https://doi.org/10.1007/s11760-020-01833-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-020-01833-x

Keywords

Search

Navigation