A better way of extracting dominant colors using salient objects with semantic segmentation
Introduction
Playing a vital role in designs, color has been used by artists and designers for decades. Color is a key element for professional design. Color combinations can affect emotions, psychology and user experience since each color in the combination has a unique effect on each other and the balance of the combination has significant importance (Machajdik and Hanbury, 2010, Terwogt and Hoeksma, 1995). Furthermore, the majority of people match colors with an intuitions developed at a young age. As there is no universally accepted combination of colors, it is a very challenging task to determine popular combinations of colors to be preferred.
Finding the right color combination is crucial for each and every aspect of our lives from designing a new product or combining clothes to create the perfect combination to decorating a space either at home or in the office. However, there is no single right color combination but rather preferred or popular color combinations. In this paper, we present a novel approach that finds the most significant dominant colors of the salient object in an image containing an extensive number of different objects either separate from or overlapped the salient object in question. Thus, our approach enables people to perceive the most suitable combination of colors for a given occasion. Designing a new product or dressing for a special event could be considered as an example for this occasion. Furthermore, when our approach is applied to images of different categories, including fashion, fragrance, interior design, etc., the spatial, temporal, and cultural variations of popular color combinations in these categories could be discerned. This data could also be used to predict the future color combination preferences for a given category e.g. determining a country level color preference for a certain type of object, or yearly most popular colors, etc.
It is common to have more than one object in an image, unless there is a special shooting setup for photographs. Moreover, whether professional or casual, in a there is always a background in a photograph. First of all we segmentate the image via semantic segmentation to obtain different objects in order to determine the salient object. Then, the original image is submitted to salient object detection and the outputs of this stage are intersected with the segmented image from the previous step. This process gives us the salient object as a whole. Finally, the colors of the salient object are quantized and then clustered to determine the dominant colors. Our contributions are as follows:
- •
we have extracted the most accurate dominant colors of the focused object in a given image when compared to existing research focusing primarily on extracting the dominant colors of either the entire image or the foreground only.
- •
we successfully applied deep learning architectures in order to semantically understand and detect the focused object in a given image.
The rest of the manuscript is organized as follows: Related Work is given in Section 2, both of the models developed for our novel approach is introduced in Section 3, Section 4 is used to represent Experimental Results with the Conclusion given in Section 5.
Section snippets
Related work
A review of related work on dominant color extraction reveals the fact that this method has been mainly used for color theme extraction and content-based image retrieval. Salient regions and salient features are only mention on Liu et al. (2018) and Xing et al. (2018) respectively. However, they do not perform an independent salient object detection as both research focuses on images containing a single object. Therefore, the aforementioned saliency only refers to the most notable regions
System design
The main motivation of this study is to extract the dominant colors of a salient object from an image even if the objects overlap each other. It is obvious that the dominant colors of an image do not always represent the dominant colors of the salient object within the image. Therefore, methods proposed in the related work section is mostly limited to dominant colors extraction of the entire image. It is evident that objects in an image must be segmentated before extracting the dominant colors.
Experimental results
In this section, we demonstrated and put forward the experiments carried out to ascertain the performance of our proposed approach. We used two different datasets to obtain the experimental results. The first dataset is the MIT Scene Parsing dataset, which was used for semantic segmentation (Zhou et al., 2017). The second dataset is a custom dataset provided by HueData (Lechner and Harrington, 0000). It was used to evaluate the performance of our dominant colors detection approach. The MIT
Conclusion
In this paper, we have investigated the problem of extracting the dominant colors of the focused object in a given image and successfully presented a novel approach for the dominant color extraction. Our approach is primarily based on semantic segmentation complemented by salient object detection. Our key contribution is the extraction of the most representative dominant colors of the salient object in an image in contrast to the current research, which produces the dominant colors treating the
CRediT authorship contribution statement
Ayse Bilge Gunduz: Conceptualization, Methodology, Software, Visualization, Writing - review & editing. Berk Taskin: Methodology, Software, Visualization. Ali Gokhan Yavuz: Writing - review & editing, Supervision. Mine Elif Karsligil: Supervision.
Declaration of Competing Interest
The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Name: A. Bilge GUNDUZ Address: Davutpasa Mah. Yildiz Technical University, Department of Computer Engineering 34220 Esenler/Istanbul/Turkey Phone: +90 554 666 18 90 E-mail: [email protected]
Acknowledgments
We would like to thank the HUEDATA for providing us with necessary datasets. Special thanks to VRAY GUILLAUME MARC GEORGES and M. YASIN SAGLAM for their contribution in running the experimental tests.
References (57)
- et al.
Finding intrinsic color themes in images with human visual perception
Neurocomputing
(2018) - et al.
Quaternion polar harmonic Fourier moments for color images
Inform. Sci.
(2018) - et al.
A fast MPEG-7 dominant color extraction with new similarity measure for image retrieval
J. Vis. Commun. Image Represent.
(2008) - et al.
Fashion recommendations through cross-media information retrieval
J. Vis. Commun. Image Represent.
(2019) The Kendall rank correlation coefficient
- et al.
k-means++: The Advantages of Careful SeedingTech. Rep.
(2006) - et al.
Segnet: A deep convolutional encoder-decoder architecture for image segmentation
IEEE Trans. Pattern Anal. Mach. Intell.
(2017) - et al.
Color image segmentation for medical images using L* a* b* color space
IOSR J. Electron. Commun. Eng.
(2012) Modeling scene illumination colour for computer vision and image reproduction: A survey of computational approaches
Comput. Sci. Simon Fraser Univ.
(1998)- Bylinskii, Z., Judd, T., Borji, A., Itti, L., Durand, F., Oliva, A., Torralba, A., 0000. MIT saliency benchmark....
Digital Image Fundamentals: Introduction to Signal and Image Processing
Dominant and LBP-based content image retrieval using combination of color, shape and texture features
Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs
IEEE Trans. Pattern Anal. Mach. Intell.
keras
A deep multi-level network for saliency prediction
The pascal visual object classes challenge 2007 results
Robust clustering using outlier-sparsity regularization
IEEE Trans. Signal Process.
Mobilenets: Efficient convolutional neural networks for mobile vision applications
Autonomous color theme extraction from images using saliency
Database architecture for content-based image retrieval
The application of cluster analysis in strategic management research: an analysis and critique
Strateg. Manag. J.
Deep gaze i: Boosting saliency prediction with feature maps trained on imagenet
Cited by (7)
Tree trunk detection in urban scenes using a multiscale attention-based deep learning method
2023, Ecological InformaticsGlobal contextually guided lightweight network for RGB-thermal urban scene understanding
2023, Engineering Applications of Artificial IntelligenceCitation Excerpt :Scene understanding is a special semantic segmentation method that performs scene object retrieval, outdoor scene segmentation, and 3D reconstruction (Cheng et al., 2021); in addition, it can be applied to remote sensing (Tang et al., 2022; Wu et al., 2022b), and medical image understanding (Zhou et al., 2022e). Moreover, it is crucial for the development of advanced driver assistance systems (Bai et al., 2022; Zhou et al., 2022a; Gunduz et al., 2021) as well as in modern robots, for performing sub-tasks, such as road recognition, pedestrian detection, or understanding traffic scenes. Urban scene understanding requires marking the whole traffic scene in the form of semantic pixels and distinguishing these into multiple categories, such as cars, buildings, pedestrians, and roadblocks.
Methodology for urban vegetation segmentation based on cartesian hue and saturation characteristics using UAV images
2022, Urban Forestry and Urban GreeningCitation Excerpt :In recent years, UAVs have presented a technology with high accessibility and high spatial resolution that has become indispensable remote sensing equipment. The use of UAVs has encouraged the development of image processing methods that aim to solve the green areas segmentation task (Gunduz et al., 2021; Juang and Wu, 2010). This novel technology enhances the implementations that use both the color and the hyperspectral information to refine the detection of the green areas (Qin et al., 2022).
Edge-aware and spectral–spatial information aggregation network for multispectral image semantic segmentation
2022, Engineering Applications of Artificial IntelligenceCitation Excerpt :It is widely used in environmental monitoring (Thomazella et al., 2018), urban planning (Saha et al., 2021), national defense security, and other fields (Kumar et al., 2020). In recent years, artificial intelligence technology represented by deep neural networks has been greatly developed in the field of image processing (Wu et al., 2021; Mou et al., 2020; Gunduz et al., 2021). In view of the particularity of remote sensing data, some researchers have begun to try to migrate the natural image semantic segmentation network model to the remote sensing image semantic segmentation task (He et al., 2022; Zhao et al., 2021).
Application Study on the Recognition of Oral Obstructed Tooth Images Using Semantic Segmentation
2023, Proceedings of SPIE - The International Society for Optical Engineering