Elsevier

Neurocomputing

Volume 412, 28 October 2020, Pages 461-476
Neurocomputing

A simulation-based few samples learning method for surface defect segmentation

https://doi.org/10.1016/j.neucom.2020.06.090Get rights and content

Abstract

In industrial production, it is difficult to obtain a well-trained surface detection algorithm since the real defect samples are lacking. In this paper, we propose a surface defect segmentation method based on defect sample simulation, which only needs few defect training samples. The entire method includes two modules: a local defect simulation algorithm and a residual-restored-based segmentation algorithm. In order to ensure both structural and local texture consistency of the simulated defects, we design a two-stage simulation algorithm based on generation adversarial net and neural style transfer. The simulation method requires one single defect reference sample for training, and can generate the same type of defect in the specified area. The segmentation algorithm, trained with the simulated images and reference samples, can restore the defect area and yield the predicted label from the residual image. We carry out experiments on the button, road crack, and silicon steel strip datasets. The results show that the proposed method can remarkably improve the defect segmentation accuracy, attaining F1 score of 0.82.

Introduction

Surface defect detection has always been an important part of industrial production, which is used to quantify the impact of the manufacturing process on product quality. Quantitative detection is similar to semantic segmentation, that is, the defect is regarded as the segmentation target, and the defect-free region can be regarded as the background for pixel-by-pixel classification.

Traditional methods used for defect detection [1], [2], [3] have some limitations and are only valid for specific texture backgrounds. The use of deep convolutional networks [4], [5], [6], [7] for defect segmentation shows great generalization, and one well-trained network is suitable for multiple defect types with various complex backgrounds. However, in the industrial production line, defect samples are scarce, directly training with which easily leads to over-fitting.

To deal with this problem, some researches seek to design better data augmentation methods [8], [9], [10], [11], [12]. Others make full use of the information contained in defect-free samples to design segmentation networks based on positive samples [13], [14], which can achieve better results in the simple texture background. Our previous work applies GLS-GAN [15] and DST [16] to generate simulation images. For each defect type, GLS-GAN can transplant existing defects into defect-free samples through regional paired iteration to generate high-quality defect images. But it still needs dozens of reference samples to reduce the similarity, and takes a long time to generate enough images. DST can generate different defect images with only one single reference. However, it cannot accurately generate the texture structure of the defect area.

In view of the above, we decide to investigate a tougher task: how to efficiently train the defect segmentation network using only one single defect image for each type of defect. There are two main difficulties: how to ensure the diversity of augmented training samples to avoid over-fitting, and how to ensure the effectivity of image augmentation with a complex background, which guarantees that the segmentation network can efficiently converge when trained with the augmented samples.

Based on the former work, we propose a novel Defect Segmentation with Simulation (DSS) framework. We depart the whole task into two parts, which are defect sample simulation and defect segmentation, and propose a two-stage defect simulation algorithm and a residual-restored-based segmentation algorithm respectively. In the simulation framework, we first use a regular GAN [17] structure for coarsely matching. Once trained with regional unpaired training strategy, it can conduct forward generation of corresponding defect types in areas of any specified shape. And then, we take the style transfer in DST as the second stage to merge the generated area into the defect-free images. The advantage of the two-stage algorithm is that it can disassemble the tough task of defect simulation upon a complex background texture, allowing each stage to focus on completing a single specific and easier function. Samples of the simulation algorithm are shown in Fig. 1. The paired simulated defect images along with their corresponding defect-free samples are then used to train the lightweight residual-restored-based segmentation network, which can restore the defect and further yield the prediction through the residual image. The backbones of the three networks share the same encoder-decoder structure which is easy to implement. And we are inspired by MobileNet [18] to apply depthwise separable convolution on the segmentation algorithm. Finally, we verify the proposed method in button [15], road crack [19], silicon steel strip [20], [21] datasets. In the case of using only defect-free samples and one single defect image for each type, the proposed method outperforms related algorithms in recent years. And the segmentation algorithm can achieve convergence within 30 epochs on all the three datasets. We also design an ablation experiment to further prove the improvement effect of each module.

In summary, our contributions are:

  • 1.

    We investigate a new task of training a defect segmentation algorithm with single defect reference and defect-free samples, and propose a novel Defect Segmentation with Simulation (DSS) framework to solve it.

  • 2.

    Based on our previous simulation work, we adapt regional unpaired training strategy on a regular GAN, to realize the forward generation of defects within any specified shape. In the meanwhile, we apply the style transfer in DST to blend the generated area into the defect-free samples.

  • 3.

    We design a novel lightweight residual-restored-based segmentation algorithm. And we use paired training strategy with three optimizers for further improvement of segmentation accuracy.

  • 4.

    We adopt a simple backbone network in the framework as well as apply depthwise separable convolution and multiple losses to the segmentation task for more efficient training.

Section snippets

Dataset augmentation

In the classification task, adopting translation with rotation [8], random cropping and mirroring [12], adding random noise [9] or other methods for data augmentation can achieve better results. Besides, some researches, such as [10], merge the images from the same class to generate new samples.

In the target detection task, it is necessary to consider the label image while augmenting the dataset, and the method of random clipping [11] is often used to improve the accuracy of the model.

Methods

This chapter will detail our proposed method. For the task of defect segmentation with only one single reference sample H for each defect type, we design a simulation algorithm Sim to obtain the simulated defect images S from defect-free images O as dataset augmentation. In addition, we design a defect segmentation algorithm Seg that can detect various kinds of real defects after trained with the simulated images. The framework of the entire method is shown in Fig. 2.

There are two major

Conclusion

In this paper, we propose a few-sample surface defect segmentation method based on defect sample simulation to deal with the scarcity of the real defect training samples. In order to generate various simulated samples with high quality, we use an adversarial network to reconstruct the structure of defect in the specified area of the defect-free samples, and then adopt the neural style transfer to blend the simulated defect into the background. Further, in order to make full use of the paired

CRediT authorship contribution statement

Taoran Wei: Conceptualization, Methodology, Investigation, Software, Formal analysis, Visualization, Writing - original draft. Danhua Cao: Resources, Project administration, Supervision, Writing - review & editing. Caiyun Zheng: Investigation, Validation, Formal analysis, Visualization, Writing - review & editing. Qun Yang: Methodology, Writing - review & editing, Data curation.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgement

We gratefully acknowledge the support from Department of science and technology, Hubei Provincial People’s Government. This research is a part of 2019AAA057 project.

Taoran Wei received the bachelor’s degree in School of Optical and Electronic Information from Huazhong Uni- versity of Science and Technology. He is currently pursu- ing a Ph.D. in Huazhong University of Science and Tech- nology. His main research areas are machine vision and quality inspection.

References (55)

  • H.-D. Lin

    Tiny surface defect inspection of electronic passive components using discrete cosine transform decomposition and cumulative sum techniques

    Image Vis. Comput.

    (2008)
  • E. Hoseini et al.

    Fabric Defect Detection Using Auto-Correlation Function

    Int. J. Comput. Theory Eng.

    (2013)
  • R. Ren et al.

    A Generic Deep-Learning-Based Approach for Automated Surface Inspection

    IEEE Trans. Cybern.

    (2018)
  • P. Y. Simard, D. Steinkraus, and J. C. Platt, ‘Best practices for convolutional neural networks applied to visual...
  • Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang, ‘Random Erasing Data Augmentation’, CoRR, vol. abs/1708.04896, Aug....
  • J. Lemley et al.

    Smart Augmentation Learning an Optimal Data Augmentation Strategy

    IEEE Access

    (2017)
  • W. Liu et al., ‘SSD: Single Shot MultiBox Detector’, Eur. Conf. Comput. VisionECCV, pp. 21--37, 2016, doi:...
  • S. Zagoruyko and N. Komodakis, ‘Wide Residual Networks’, ArXiv160507146 Cs, May 2016, Accessed: Jul. 23, 2019....
  • Z. Zhao et al.

    A Surface Defect Detection Method Based on Positive Samples

  • S. Mei et al.

    An Unsupervised-Learning-Based Approach for Automated Defect Inspection on Textured Surfaces

    IEEE Trans. Instrum. Meas.

    (2018)
  • T. Wei, D. Cao, X. Jiang, C. Zheng, and L. Liu, ‘Defective samples simulation through Neural Style Transfer for...
  • I. Goodfellow et al., ‘Generative Adversarial Nets’, Adv. Neural Inf. Process. Syst., pp. 2672--2680,...
  • M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, ‘MobileNetV2: Inverted Residuals and Linear Bottlenecks’,...
  • R. Amhaz et al.

    Automatic crack detection on 2D pavement images: An algorithm based on minimal path selection

    IEEE Trans. Intell. Transp. Syst.

    (2016)
  • K.-C. Song et al.

    Surface Defect Detection Method Using Saliency Linear Scanning Morphology for Silicon Steel Strip under Oil Pollution Interference

    ISIJ Int.

    (2014)
  • K. Song et al.

    Micro Surface Defect Detection Method for Silicon Steel Strip Based on Saliency Convex Active Contour Model

    Math. Probl. Eng.

    (2013)
  • B. Zoph, E. D. Cubuk, G. Ghiasi, T.-Y. Lin, J. Shlens, and Q. V. Le, ‘Learning Data Augmentation Strategies for Object...
  • Cited by (9)

    View all citing articles on Scopus

    Taoran Wei received the bachelor’s degree in School of Optical and Electronic Information from Huazhong Uni- versity of Science and Technology. He is currently pursu- ing a Ph.D. in Huazhong University of Science and Tech- nology. His main research areas are machine vision and quality inspection.

    Danhua Cao is a professor in the School of Optical and Electronic Information, Huazhong University of Science and Technology. She received her Ph.D. degree in elec- tronic physics and devices from Huazhong University of Science and Technology in 1993. She is a permanent member of the Professional Committee of Opto-electronic Technology in the Chinese Optical Society. Her research interests include optoelectronic sensing and signal pro- cessing as well as machine vision algorithms and systems

    Caiyun Zheng received the bachelor's degree in School of Optical and Electronic Information from Huazhong University of Science and Technology. She is currently studying for a master's degree at the Huazhong University of Science and Technology. Her research interests include machine vision, deep learning and automated surface inspection

    Qun Yang received his B.E. degree in 2017 from the School of Optical and Electronic Information, Huazhong University of Science and Technology, where he is currently pursuing his M.E. degree. He is interested in the field of image processing, machine learning.

    View full text