Elsevier

Applied Soft Computing

Volume 98, January 2021, 106786
Applied Soft Computing

An automatic coloring method for ethnic costume sketches based on generative adversarial networks

https://doi.org/10.1016/j.asoc.2020.106786Get rights and content

Highlights

  • Based on the color characteristics and styles of national costumes, this paper aims to establish a neural network automatic coloring model to achieve a stable, efficient and automatic coloring of the sketches of ethnic costumes.

  • It provides an effective means for the digitalization of minority apparel culture, and promote the promotion and application of color elements of minority apparel in auxiliary costume design and other related fields.

Abstract

In this paper, we present an automatic coloring model for ethnic costume sketches based on a generative adversarial networks. The proposed model is composed of a 6-layer U-net structure generator and a discriminator with 5-layer convolutional neural network. Then the loss function and the weights of true reliability value in the discriminator are optimized. And, finally the constructed sketch database is used for training to get an automatic coloring model. The results of a large number of sketch coloring experiments show that the model has good learning ability on exploring color law of ethnic costumes and can achieve better coloring effects.

Introduction

Ethnic minority costumes have distinctive characteristics such as bright colors, high contrast, and many patterns. Especially the geometric shapes, animal and plant patterns, totems, etc. Contained in the costumes have a variety of aesthetic symbols, which is one of the important manifestations of national culture. With the rapid development of multimedia technology, the needs of ethnic costumes in academic research, art design and other fields are also expanding. In this paper we introduce an automatic coloring model for ethnic costume sketches which is no one has done before. More and more scholars have begun to pay attention to the protection and inheritance of traditional minority cultures, under which background digital processing and application is undoubtedly one of the effective ways.

Currently, there are related researches on the digital processing and application of ethnic costumes, such as image segmentation, matching and retrieval of ethnic costumes [1], [2], [3]. But these methods have some shortcomings such as low degree of technical application and dependence on professional experience, and the analysis of cultural resources of national costumes is limited in shallow feature extraction and content search, etc. In the application of national costume culture that relies on aesthetic knowledge and personal innovation, there are no further explorations or key technologies for digitizing ethnic costume Development and utilization, such as discovering and incorporating the colors and elements of national costumes in graphic design and handicraft creation.

Hand-drawn sketches may be the only method available for everyone to describe things [4], and one of the simplest and most intuitive ways for humans to communicate and store information. In recent years, with the rapid development of finger touch screens and mobile terminal devices, people can easily get hand-drawn sketches by sliding their fingers and using a stylus, which has brought academia and industry attention to hand-drawn sketches, including image retrieval based on hand-drawn sketches (Sketch-based image retrieval, SBIR) [5], [6], [7], image reconstruction based on hand-drawn sketches [8], product design [9], etc. In the field of costume design, sketching painting art has always been used to express design concepts and inspiration, and color serves as an important part of the national costume culture. Therefore, the coloring of costume sketch is a crucial link in the costume design process, so as to realize the application of the color law of minority costumes.

Traditional feature extraction based image coloring and matching methods include those who are based on color transfer, color expansion and image segmentation. With the development of deep neural networks in recent years, especially the powerful capabilities of convolutional neural networks in the field of image processing, more and more researchers use deep neural networks for coloring. The current best methods for image coloring are basically based on generative adversarial networks. Existing researchers have performed image coloring on maps [10], comics [11], and architectural images [12], but most of their research focuses on fashion, with inadequate attention on coloring hand-drawn sketches of ethnic costume images. Therefore, based on the color characteristics and styles of ethnic costumes, in this paper we firstly established a sketch library of ethnic costumes for costume coloring tasks, and also designed an end-to-end neural network automatic coloring model for ethnic costume sketch using GAN. The proposed approach can achieve a stable, efficient and automatic coloring of the sketches of ethnic costumes, which will provide an effective means for the digitalization of minority apparel culture, and promote the promotion and application of color elements of minority apparel in auxiliary costume design and other related fields.

Section snippets

Related work

Generative adversarial networks (GAN), first proposed in 2014, is a generative deep learning model that integrates antagonistic thoughts [13]. In the years after the proposal, it has rapidly become a research hotspot in the field of AI, and new models and theories have been put forward. GAN can simulate the distribution of target data and generate a large number of non-human annotated data, which is timely for deep learning. In addition, GAN plays a unique role in the fields of reality

Frame construction of the proposed model

The input training data of a pair of generative adversarial networks can be obtained by matching the hand-drawn sketch library with the original image one by one. However, as part of data sets, in order to find the underlying laws, it is necessary to use deep neural network. No matter what kind of network structure it is, its essence is to infer unknown properties based on known properties. In the field of deep learning, the less the characteristics and participation of artificial design are,

Building and training of the proposed model

The following describes the construction of the ethnic costume coloring network from the aspects of generator network structure, discriminator network structure, activation function, loss function, etc.

Model training

The model training of generative adversarial networks involves two independent parts: generator and discriminator. In this paper, adaptive moment estimation optimizer Adam is used to train the network. The learning rate of each parameter is inversely proportional to its previous history accumulation and sum. That is to say, the more the previous movement is, the more careful the movement is, and the less the previous movement is, the larger the movement is. At the beginning, the discriminator

Experimental setup

(1) Data set. There are few existing studies on the coloring of minority costume sketches, and there is no standard data set. We have a large number of Va costumes images, and the color patterns of the Va costumes images are obvious, which is convenient for comparative analysis of experiments. So in this paper, some edge detection algorithms are used to generate costume sketch database, in which there are 740 pairs of Va costume patterns and corresponding ethnic costume sketches in different

Conclusion

In this paper, we studied the problem of coloring ethnic costume sketches, proposed an automatic coloring method of ethnic costume sketches based on generative adversarial networks, and designs and built a coloring model of ethnic costume sketches based on generative adversarial networks. The model uses smooth loss to increase the stability during training by adjusting the network structure. It uses a fully connected layer in the output layer to reduce human intervention on the parameters. Then

CRediT authorship contribution statement

Bo Liu: Writing - original draft, Data curation. Jianhou Gan: Conceptualization, Methodology. Bin Wen: Software, Validation, Investigation. Yiping LiuFu: Visualization. Wei Gao: Supervision, Writing - review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work is supported by the National Natural Science Foundation of China under Grant No. 61862068, Yunnan Expert Workstation of Xiaochun Cao, China, and Kunming Key Laboratory of Education Information, China .

References (24)

  • LiuY. et al.

    Auto-painter: Cartoon image generation from sketch by using conditional Wasserstein generative adversarial networks

    Neurocomputing

    (2018)
  • ZhangX. et al.

    Better freehand sketch synthesis for sketch-based image retrieval: Beyond image edges

    Neurocomputing

    (2018)
  • WangY. et al.

    Application of improved fuzzy C-means algorithm for ethnic costume image segmentation

    Comput. Eng.

    (2017)
  • HouX. et al.

    The national costume pattern elements segmentation by incorporating morphology connected component and CV model

    J. Zhejiang Univ. (Sci. Ed.)

    (2019)
  • ZhouJ. et al.

    Exploiting best practice of deep CNNs features for national costume image retrieval

    Int. J. Perform. Eng.

    (2018)
  • EitzM. et al.

    How do humans sketch objects?

    ACM Trans. Graph.

    (2012)
  • LiY. et al.

    A survey of sketch-based image retrieval

    Mach. Vis. Appl.

    (2018)
  • XuD. et al.

    Cross-paced representation learning with partial curricula for sketch-based image retrieval

    IEEE Trans. Image Process.

    (2018)
  • EitzM. et al.

    Sketch-based image retrieval: benchmark and bag-of-features descriptors

    IEEE Trans. Vis. Comput. Graph.

    (2011)
  • L. Wang, V.A. Sindagi, V.M. Patel, High-quality facial photo-sketch synthesis using multi-adversarial networks, in:...
  • Z. Lu, H. Yuan, W. Bi, et al. Research on sketch-based design, in: 2009 IEEE 10th International Conference on...
  • P. Isola, J.Y. Zhu, T.H. Zhou, et al. Image-to-Image translation with conditional adversarial networks, in: Computer...
  • Cited by (17)

    • Fast style transfer for ethnic patterns innovation

      2024, Expert Systems with Applications
    • Defining a deep neural network ensemble for identifying fabric colors

      2022, Applied Soft Computing
      Citation Excerpt :

      It is also a relevant topic in several research fields, for example in physics to model light and reflectance, in physiology to model perception, in biology to model the visual system, in art to model beauty. As a result, the study of colors has fascinated, and continues to fascinate, researchers from a wide variety of fields [1,2]. One of the general problems with colors concerns the identification of the color of objects and, more generally, of any entity.

    • A generative adversarial neural network model for industrial boiler data repair

      2021, Applied Soft Computing
      Citation Excerpt :

      In computer image research, Meng Zhang and Youyi Zheng have realized the generation of a three-dimensional structure image from a two-dimensional hair structure [26]. Based on the color characteristics and styles of national costumes, Bo Liu and Jianhou Gan established a generative adversarial networks automatic coloring model to achieve a stable, efficient and automatic coloring of the sketches of ethnic costumes [27]. In order to remove mixed gaussian-impulse noise from image, one three-part Generative Adversarial Network (GAN) was proposed.

    View all citing articles on Scopus
    View full text