Specular highlight region restoration using image clustering and inpainting☆,☆☆
Introduction
A specular highlight effect is a critical disturbance in the camera image [1], [2], [3], [4]. For this reason, there have been several existing techniques for removing specular highlight focusing only on facial images [5] or handling the situations where specular highlight is weak [6], [7], [8], [9], [10]. For effective removal of specular highlight in general, the scheme should be effective for any region and material (see Fig. 1).
Also, the specular highlight region should be distinguished from the white objects in the theory of color homeostasis [11], [12]. As a cornerstone of research to solve specular highlight in general, processing algorithms for detecting specular highlight regions by designating specific threshold values or threshold ranges have been introduced [23], [24], [25], [26]. Recently owing to the development of the GPU (Graphics Processing Unit), a recovery algorithm has been introduced after the detection process in a general situation [48]. In this case, the highlighted region is predominantly detected, which results in performance deterioration in complex pattern images. Also, its application is limited to only a single image, not real-time. Alternatively, there is a method where the specular highlight regions are detected based on machine learning rather than the threshold method [22], [27]. This scheme requires a large amount of computation. It is not able to differentiate a white object from a specular highlight (see Fig. 2).
The proposed system has two major parts: detection and inpainting. Frist, the proposed specular highlight detection system classifies the specular highlight in the HSI color space. HSI color space has better visual consistency, which is more suitable for human judgment to the naked eye [13]. Therefore, HSI color space has been used in many papers for image processing [14], [15], [16], [17], [18]. The converted HSI image is applied to the newly defined classification table to detect specular highlight. This table statistically summarized the H, S, and I value of the specular highlight region by applying the proposed clustering algorithm to a lot of images. The proposed clustering algorithm is consisting of the K-means clustering, newly defined filters, and classification equations.
In the case of restoration part, we apply conventional inpainting methods. Image inpainting is generally defined as the process of restoring missing pixels and damaged regions [19]. Furthermore, various conventional image inpainting method already perform high-quality performance. In this paper, two inpainting techniques were used with proposed detection algorithm to compare performance. Therefore, the proposed detection algorithm has versatility because it can be fused with various inpainting techniques used in many different environments.
In summary, the following are the main contributions of this work:
- •
We propose a specular highlight region restoration method based on proposed image clustering algorithm and conventional inpainting architectures.
- •
The clustering process made the classification table for detecting specular highlight. This table is remarkably simple and easy to use. Therefore, it can be obtained superiority in processing time.
- •
By adding proposed image clustering algorithm to the various type image inpainting architecture for various industries, it can be obtained the superiority in speed and efficiency. This means that the image clustering algorithm is versatile.
Section snippets
Related work
There are two primary approaches for detecting specular highlight. The first approach is to detect specular highlight using the threshold value, and the second approach is specular detection through preprocessing definition.
Specular highlight region model
In this study, HSI color space based specular highlight detection approach is considered. Initially, the reflection effect of highlight acting on any point of the object according to the Lambert model is defined as follows [28], [29].where, is any point of the object, is the sum of the reflection intensities of the highlight acting on the object. Further, , , and are the intensities
Specular highlight region detection
The proposed specular highlight detection algorithm is based on the classification table. This table is constructed by an algorithm of Table 1. A total of 1000 images consisting of 300 single images and 700 real-time images are entered. Among them, 200 single images and 300 real-time images are sampled.
In this algorithm, the clustering process is included to search the regions that are suspected to be specular highlights. First of all, the input image is converted to HSI image. And then HSI
Image inpainting for specular highlight region restoration
Image inpainting technique is used to restore a detected specular highlight region [19], [20], [21], [32], [33], [34], [35], [36], [37], [40], [41], [42], [43], [44], [45], [46]. Amount of image data is required for general-purpose results that can be used in a real-time environment. Consequently, this study used a total of 1,800,000 image data using the Image Inpainting software from NVIDIA [38]; further, as a random masking image data to be used in the training process, an 8-bit specular
Comparison of the detection results
An experiment was conducted to determine the effectiveness of the defined data. Fig. 8 shows an example of the detection results in comparison with that of the four conventional methods. Examples of images used in the experiment included two single images with specular highlight, and one image with specular highlight captured in the real-time environment. Further, in the real-time environment, the image captured with various objects had a specular highlight on the whiteboard. In other words, it
Conclusion
In this research, the specular light region has been detected using the classification table which is the pre-processing pixel clustering data. The pixel clustering data are clustered through three newly defined filter equations and two classification equations, and the common pixel values of the clustered models are classified according to light intensity and quantified. Through the experiments, it is demonstrated that the detection of specular highlight regions is faster and more accurate
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
This research was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No.2019R1A2C2088859).
This research was supported by the Lunar Exploration Program through the KARI grant funded by the Ministry of science and ICT (No.SR18040).
References (49)
- et al.
Classification of farmland images based on color features
J. Visual Commun. Image Representation (JVCIR)
(2015) - et al.
A survey on image data analysis through clustering techniques for real world applications
J. Visual Commun. Image Representation (JVCIR)
(2018) - et al.
A critical survey of state-of-the-art image inpainting quality assessment metrics
J. Visual Commun. Image Representation (JVCIR)
(2017) - et al.
Image segmentation using K-means clustering algorithm and subtractive clustering algorithm
Procedia Comput. Sci.
(2015) - et al.
Image alignment method based on CUDA SURF for multi-spectral machine vision application
J. Korea Multimedia Soc.
(2014) - et al.
Surface inspection of a journal bearing using machine vision
J. Korean Soc. Precis. Eng.
(2017) - et al.
Gaze tracking using a modified strarburst algorithm and homography normalization
J. Korea Inst. Inf. Commun. Eng.
(2014) - et al.
Information fusion of cameras and laser radars for perceoption systems of autonomous vehicles
J. Korean Inst. Intelligent Syst.
(2013) - L.O. Solis Sánchez, J.J. García Escalante, R. Castañeda Miranda, I. Torres Pacheco, Chen Li, S. Lin, K. Zhou, K....
- et al.
Real-time specular highlight removal using bilateral filtering
ECCV Part IV LNCS
(2010)
Real-time highlight removal using intensity ratio
Appl. Opt.
Real-time high-quality specular highlight removal using efficient pixel clustering
SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI)
Specular highlight removal for real-world images
Comput. Graphics Forum
Single image reflection removal using deep encoder-decoder network
Conference on CVPR (Computer Vision and pattern Recognioition)
Color cast detection based on color by correlation and color constancy algorithm using kernel density estimation
J. Korea Multimedia Soc.
A comparison of computational color constancy algorithms Part II: Experiments with image data
IEEE Trans. Image Process.
Dehazing in HSI color space with color correction
JBE
Constructing cylindrical coordinate color spaces
Pattern Recogn. Lett.
Generative Adversarial Nets
A truncated least squares approach to the detection of specular highlights in color images
In International Conference on Robotics and Automation (ICRA)
Removal of specular reflections in endoscopic images
Acta Polytechnica J. Adv. Eng.
Cited by (4)
A Method for Detecting and Correcting Specular Highlights in Capsule Endoscope Images Based on Independent Cluster Distribution
2023, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)Improved two-stage image inpainting with perceptual color loss and modified region normalization
2022, Machine Vision and ApplicationsA modified gamma correction algorithm applied to the medium and bright light problems in AI image recognition
2021, Journal of Physics: Conference Series
- ☆
This paper has been recommended for acceptance by S. Sarkar.
- ☆☆
Abbreviations: Pixel Analyzing; Pixel Clustering; Specular Highlight identification, Detection; Image Inpainting; Generative Adversarial Net.