A better way of extracting dominant colors using salient objects with semantic segmentation

https://doi.org/10.1016/j.engappai.2021.104204Get rights and content

Abstract

One of the most prominent parts of professional design consists of combining the right colors. This combination can affect emotions, psychology, and user experience since each color in the combination has a unique effect on each other. It is a very challenging to determine the combination of colors since there are no universally accepted rules for it. Yet finding the right color combination is crucial when it comes to designing a new product or decorating the interiors of a room. The main motivation of this study is to extract the dominant colors of a salient object from an image even if the objects overlap each other. In this way, it is possible to find frequent and popular color combinations of a specific object. So, first of all, a modified Inception-ResNet architecture was designed semantically segmentate objects in the image. Then, SALGAN was applied to find the salient object in the image since the aim here is to find the dominant colors of the salient object in a given image. After that, the outputs consisted of the SALGAN applied image and segmented image were combined to obtain the corresponding segment for the purpose of finding the salient object on the image. Finally, since we aimed to quantize the pixels of the corresponding segment in the image, we applied k-means clustering which partitions samples into K clusters. The algorithm works iteratively to assign each data point to one of the K groups based on their features. Data points were clustered according to feature similarity. As a result the clustering, the most relevant dominant colors were extracted. Our comprehensive experimental survey has demonstrated the effectiveness of the proposed method.

Introduction

Playing a vital role in designs, color has been used by artists and designers for decades. Color is a key element for professional design. Color combinations can affect emotions, psychology and user experience since each color in the combination has a unique effect on each other and the balance of the combination has significant importance (Machajdik and Hanbury, 2010, Terwogt and Hoeksma, 1995). Furthermore, the majority of people match colors with an intuitions developed at a young age. As there is no universally accepted combination of colors, it is a very challenging task to determine popular combinations of colors to be preferred.

Finding the right color combination is crucial for each and every aspect of our lives from designing a new product or combining clothes to create the perfect combination to decorating a space either at home or in the office. However, there is no single right color combination but rather preferred or popular color combinations. In this paper, we present a novel approach that finds the most significant dominant colors of the salient object in an image containing an extensive number of different objects either separate from or overlapped the salient object in question. Thus, our approach enables people to perceive the most suitable combination of colors for a given occasion. Designing a new product or dressing for a special event could be considered as an example for this occasion. Furthermore, when our approach is applied to images of different categories, including fashion, fragrance, interior design, etc., the spatial, temporal, and cultural variations of popular color combinations in these categories could be discerned. This data could also be used to predict the future color combination preferences for a given category e.g. determining a country level color preference for a certain type of object, or yearly most popular colors, etc.

It is common to have more than one object in an image, unless there is a special shooting setup for photographs. Moreover, whether professional or casual, in a there is always a background in a photograph. First of all we segmentate the image via semantic segmentation to obtain different objects in order to determine the salient object. Then, the original image is submitted to salient object detection and the outputs of this stage are intersected with the segmented image from the previous step. This process gives us the salient object as a whole. Finally, the colors of the salient object are quantized and then clustered to determine the dominant colors. Our contributions are as follows:

  • we have extracted the most accurate dominant colors of the focused object in a given image when compared to existing research focusing primarily on extracting the dominant colors of either the entire image or the foreground only.

  • we successfully applied deep learning architectures in order to semantically understand and detect the focused object in a given image.

The rest of the manuscript is organized as follows: Related Work is given in Section 2, both of the models developed for our novel approach is introduced in Section 3, Section 4 is used to represent Experimental Results with the Conclusion given in Section 5.

Section snippets

Related work

A review of related work on dominant color extraction reveals the fact that this method has been mainly used for color theme extraction and content-based image retrieval. Salient regions and salient features are only mention on Liu et al. (2018) and Xing et al. (2018) respectively. However, they do not perform an independent salient object detection as both research focuses on images containing a single object. Therefore, the aforementioned saliency only refers to the most notable regions

System design

The main motivation of this study is to extract the dominant colors of a salient object from an image even if the objects overlap each other. It is obvious that the dominant colors of an image do not always represent the dominant colors of the salient object within the image. Therefore, methods proposed in the related work section is mostly limited to dominant colors extraction of the entire image. It is evident that objects in an image must be segmentated before extracting the dominant colors.

Experimental results

In this section, we demonstrated and put forward the experiments carried out to ascertain the performance of our proposed approach. We used two different datasets to obtain the experimental results. The first dataset is the MIT Scene Parsing dataset, which was used for semantic segmentation (Zhou et al., 2017). The second dataset is a custom dataset provided by HueData (Lechner and Harrington, 0000). It was used to evaluate the performance of our dominant colors detection approach. The MIT

Conclusion

In this paper, we have investigated the problem of extracting the dominant colors of the focused object in a given image and successfully presented a novel approach for the dominant color extraction. Our approach is primarily based on semantic segmentation complemented by salient object detection. Our key contribution is the extraction of the most representative dominant colors of the salient object in an image in contrast to the current research, which produces the dominant colors treating the

CRediT authorship contribution statement

Ayse Bilge Gunduz: Conceptualization, Methodology, Software, Visualization, Writing - review & editing. Berk Taskin: Methodology, Software, Visualization. Ali Gokhan Yavuz: Writing - review & editing, Supervision. Mine Elif Karsligil: Supervision.

Declaration of Competing Interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Name: A. Bilge GUNDUZ Address: Davutpasa Mah. Yildiz Technical University, Department of Computer Engineering 34220 Esenler/Istanbul/Turkey Phone: +90 554 666 18 90 E-mail: [email protected]

Acknowledgments

We would like to thank the HUEDATA for providing us with necessary datasets. Special thanks to VRAY GUILLAUME MARC GEORGES and M. YASIN SAGLAM for their contribution in running the experimental tests.

References (57)

  • CattinP.

    Digital Image Fundamentals: Introduction to Signal and Image Processing

    (2016)
  • ChauhanS. et al.

    Dominant and LBP-based content image retrieval using combination of color, shape and texture features

  • ChenL.-C. et al.

    Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2017)
  • Cheng, M.-M., Warrell, J., Lin, W.-Y., Zheng, S., Vineet, V., Crook, N., 2013. Efficient salient region detection with...
  • Chollet, F., 2017. Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE...
  • CholletF.

    keras

    (2015)
  • CorniaM. et al.

    A deep multi-level network for saliency prediction

  • EveringhamM. et al.

    The pascal visual object classes challenge 2007 results

    (2007)
  • ForeroP.A. et al.

    Robust clustering using outlier-sparsity regularization

    IEEE Trans. Signal Process.

    (2012)
  • He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition. In: Proceedings of the IEEE...
  • HowardA.G. et al.

    Mobilenets: Efficient convolutional neural networks for mobile vision applications

    (2017)
  • Huang, X., Shen, C., Boix, X., Zhao, Q., 2015. Salicon: Reducing the semantic gap in saliency prediction by adapting...
  • JahanianA. et al.

    Autonomous color theme extraction from images using saliency

  • Jetley, S., Murray, N., Vig, E., 2016. End-to-end saliency mapping via probability distribution prediction. In:...
  • KatoT.

    Database architecture for content-based image retrieval

  • KetchenD.J. et al.

    The application of cluster analysis in strategic management research: an analysis and critique

    Strateg. Manag. J.

    (1996)
  • KümmererM. et al.

    Deep gaze i: Boosting saliency prediction with feature maps trained on imagenet

    (2014)
  • Kummerer, M., Wallis, T.S., Gatys, L.A., Bethge, M., 2017. Understanding low-and high-level contributions to fixation...
  • Cited by (7)

    • Global contextually guided lightweight network for RGB-thermal urban scene understanding

      2023, Engineering Applications of Artificial Intelligence
      Citation Excerpt :

      Scene understanding is a special semantic segmentation method that performs scene object retrieval, outdoor scene segmentation, and 3D reconstruction (Cheng et al., 2021); in addition, it can be applied to remote sensing (Tang et al., 2022; Wu et al., 2022b), and medical image understanding (Zhou et al., 2022e). Moreover, it is crucial for the development of advanced driver assistance systems (Bai et al., 2022; Zhou et al., 2022a; Gunduz et al., 2021) as well as in modern robots, for performing sub-tasks, such as road recognition, pedestrian detection, or understanding traffic scenes. Urban scene understanding requires marking the whole traffic scene in the form of semantic pixels and distinguishing these into multiple categories, such as cars, buildings, pedestrians, and roadblocks.

    • Methodology for urban vegetation segmentation based on cartesian hue and saturation characteristics using UAV images

      2022, Urban Forestry and Urban Greening
      Citation Excerpt :

      In recent years, UAVs have presented a technology with high accessibility and high spatial resolution that has become indispensable remote sensing equipment. The use of UAVs has encouraged the development of image processing methods that aim to solve the green areas segmentation task (Gunduz et al., 2021; Juang and Wu, 2010). This novel technology enhances the implementations that use both the color and the hyperspectral information to refine the detection of the green areas (Qin et al., 2022).

    • Edge-aware and spectral–spatial information aggregation network for multispectral image semantic segmentation

      2022, Engineering Applications of Artificial Intelligence
      Citation Excerpt :

      It is widely used in environmental monitoring (Thomazella et al., 2018), urban planning (Saha et al., 2021), national defense security, and other fields (Kumar et al., 2020). In recent years, artificial intelligence technology represented by deep neural networks has been greatly developed in the field of image processing (Wu et al., 2021; Mou et al., 2020; Gunduz et al., 2021). In view of the particularity of remote sensing data, some researchers have begun to try to migrate the natural image semantic segmentation network model to the remote sensing image semantic segmentation task (He et al., 2022; Zhao et al., 2021).

    • Application Study on the Recognition of Oral Obstructed Tooth Images Using Semantic Segmentation

      2023, Proceedings of SPIE - The International Society for Optical Engineering
    View all citing articles on Scopus
    View full text