Specular highlight region restoration using image clustering and inpainting,☆☆

https://doi.org/10.1016/j.jvcir.2021.103106Get rights and content

Highlights

  • We restore a specular highlight region based on an image clustering process.

  • The clustering process made the simple classification table to detect highlight.

  • It is good compatibility with various conventional image painting techniques.

  • The proposed method can be applied quickly and easily in various environments.

Abstract

A novel specular highlight restoration algorithm has been proposed to remove the specular highlight in a real-time vision system. In this paper, the specular highlight region has been detected and it has been restored by image inpainting method. Most of specular highlight detection algorithms are effective only for a single image. And the auto-threshold algorithm has been implemented in real time, but, it has still low reliability with a heavy computational cost. The proposed system detects pixels corresponding to the specular highlight region in the HSI color space through a newly defined classification table. So, Specular highlight can be detected quickly with these simplified classification table. In addition, it has versatility because it can be combined with various existing high-performance image inpainting method. The superiority of the proposed algorithm is compared with the conventional specular highlight removal algorithm by combining proposed algorithm with two high-performance image inpainting techniques.

Introduction

A specular highlight effect is a critical disturbance in the camera image [1], [2], [3], [4]. For this reason, there have been several existing techniques for removing specular highlight focusing only on facial images [5] or handling the situations where specular highlight is weak [6], [7], [8], [9], [10]. For effective removal of specular highlight in general, the scheme should be effective for any region and material (see Fig. 1).

Also, the specular highlight region should be distinguished from the white objects in the theory of color homeostasis [11], [12]. As a cornerstone of research to solve specular highlight in general, processing algorithms for detecting specular highlight regions by designating specific threshold values or threshold ranges have been introduced [23], [24], [25], [26]. Recently owing to the development of the GPU (Graphics Processing Unit), a recovery algorithm has been introduced after the detection process in a general situation [48]. In this case, the highlighted region is predominantly detected, which results in performance deterioration in complex pattern images. Also, its application is limited to only a single image, not real-time. Alternatively, there is a method where the specular highlight regions are detected based on machine learning rather than the threshold method [22], [27]. This scheme requires a large amount of computation. It is not able to differentiate a white object from a specular highlight (see Fig. 2).

The proposed system has two major parts: detection and inpainting. Frist, the proposed specular highlight detection system classifies the specular highlight in the HSI color space. HSI color space has better visual consistency, which is more suitable for human judgment to the naked eye [13]. Therefore, HSI color space has been used in many papers for image processing [14], [15], [16], [17], [18]. The converted HSI image is applied to the newly defined classification table to detect specular highlight. This table statistically summarized the H, S, and I value of the specular highlight region by applying the proposed clustering algorithm to a lot of images. The proposed clustering algorithm is consisting of the K-means clustering, newly defined filters, and classification equations.

In the case of restoration part, we apply conventional inpainting methods. Image inpainting is generally defined as the process of restoring missing pixels and damaged regions [19]. Furthermore, various conventional image inpainting method already perform high-quality performance. In this paper, two inpainting techniques were used with proposed detection algorithm to compare performance. Therefore, the proposed detection algorithm has versatility because it can be fused with various inpainting techniques used in many different environments.

In summary, the following are the main contributions of this work:

  • We propose a specular highlight region restoration method based on proposed image clustering algorithm and conventional inpainting architectures.

  • The clustering process made the classification table for detecting specular highlight. This table is remarkably simple and easy to use. Therefore, it can be obtained superiority in processing time.

  • By adding proposed image clustering algorithm to the various type image inpainting architecture for various industries, it can be obtained the superiority in speed and efficiency. This means that the image clustering algorithm is versatile.

Section snippets

Related work

There are two primary approaches for detecting specular highlight. The first approach is to detect specular highlight using the threshold value, and the second approach is specular detection through preprocessing definition.

Specular highlight region model

In this study, HSI color space based specular highlight detection approach is considered. Initially, the reflection effect of highlight acting on any point of the object according to the Lambert model is defined as follows [28], [29].Ip(x,y)=Ip(x,y)specular+Ip(x,y)diffuse+Ip(x,y)ambientwhere, p(x,y) is any point of the object, Ip(x,y) is the sum of the reflection intensities of the highlight acting on the object. Further, Ip(x,y)specular, Ip(x,y)diffuse, and Ip(x,y)ambientare the intensities

Specular highlight region detection

The proposed specular highlight detection algorithm is based on the classification table. This table is constructed by an algorithm of Table 1. A total of 1000 images consisting of 300 single images and 700 real-time images are entered. Among them, 200 single images and 300 real-time images are sampled.

In this algorithm, the clustering process is included to search the regions that are suspected to be specular highlights. First of all, the input image is converted to HSI image. And then HSI

Image inpainting for specular highlight region restoration

Image inpainting technique is used to restore a detected specular highlight region [19], [20], [21], [32], [33], [34], [35], [36], [37], [40], [41], [42], [43], [44], [45], [46]. Amount of image data is required for general-purpose results that can be used in a real-time environment. Consequently, this study used a total of 1,800,000 image data using the Image Inpainting software from NVIDIA [38]; further, as a random masking image data to be used in the training process, an 8-bit specular

Comparison of the detection results

An experiment was conducted to determine the effectiveness of the defined data. Fig. 8 shows an example of the detection results in comparison with that of the four conventional methods. Examples of images used in the experiment included two single images with specular highlight, and one image with specular highlight captured in the real-time environment. Further, in the real-time environment, the image captured with various objects had a specular highlight on the whiteboard. In other words, it

Conclusion

In this research, the specular light region has been detected using the classification table which is the pre-processing pixel clustering data. The pixel clustering data are clustered through three newly defined filter equations and two classification equations, and the common pixel values ​​of the clustered models are classified according to light intensity and quantified. Through the experiments, it is demonstrated that the detection of specular highlight regions is faster and more accurate

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This research was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No.2019R1A2C2088859).

This research was supported by the Lunar Exploration Program through the KARI grant funded by the Ministry of science and ICT (No.SR18040).

References (49)

  • H.L. Shen et al.

    Real-time highlight removal using intensity ratio

    Appl. Opt.

    (2013)
  • A.C.S. Souza et al.

    Real-time high-quality specular highlight removal using efficient pixel clustering

    SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI)

    (2018)
  • G. Fu et al.

    Specular highlight removal for real-world images

    Comput. Graphics Forum

    (2019)
  • Z. Chi et al.

    Single image reflection removal using deep encoder-decoder network

    Conference on CVPR (Computer Vision and pattern Recognioition)

    (2018)
  • J.W. Jung et al.

    Color cast detection based on color by correlation and color constancy algorithm using kernel density estimation

    J. Korea Multimedia Soc.

    (2010)
  • K. Barnard et al.

    A comparison of computational color constancy algorithms Part II: Experiments with image data

    IEEE Trans. Image Process.

    (2002)
  • T.H. Um et al.

    Dehazing in HSI color space with color correction

    JBE

    (2013)
  • M.A. Alonso Perez, J.J. Baez ROJAS, Conversion from n bands color space to HSI color space, Opt. Rev., 16 (2) (2009)...
  • J. Kender, Saturation, hue and normalized color: Calculation, Digitization Effects, and Use, Technical Report,...
  • H. Allan

    Constructing cylindrical coordinate color spaces

    Pattern Recogn. Lett.

    (2008)
  • R.A. Yeh, C.C., T.Y. Lim, A.G. Schwing, M.H.W. Johnson, M.N. Do, Semantic Image Inpainting with Deep Generative Models,...
  • I.J. Goodfellow et al.

    Generative Adversarial Nets

  • J.B. Park et al.

    A truncated least squares approach to the detection of specular highlights in color images

    In International Conference on Robotics and Automation (ICRA)

    (2003)
  • T. Stehle

    Removal of specular reflections in endoscopic images

    Acta Polytechnica J. Adv. Eng.

    (2006)
  • This paper has been recommended for acceptance by S. Sarkar.

    ☆☆

    Abbreviations: Pixel Analyzing; Pixel Clustering; Specular Highlight identification, Detection; Image Inpainting; Generative Adversarial Net.

    View full text