A simulation-based few samples learning method for surface defect segmentation
Introduction
Surface defect detection has always been an important part of industrial production, which is used to quantify the impact of the manufacturing process on product quality. Quantitative detection is similar to semantic segmentation, that is, the defect is regarded as the segmentation target, and the defect-free region can be regarded as the background for pixel-by-pixel classification.
Traditional methods used for defect detection [1], [2], [3] have some limitations and are only valid for specific texture backgrounds. The use of deep convolutional networks [4], [5], [6], [7] for defect segmentation shows great generalization, and one well-trained network is suitable for multiple defect types with various complex backgrounds. However, in the industrial production line, defect samples are scarce, directly training with which easily leads to over-fitting.
To deal with this problem, some researches seek to design better data augmentation methods [8], [9], [10], [11], [12]. Others make full use of the information contained in defect-free samples to design segmentation networks based on positive samples [13], [14], which can achieve better results in the simple texture background. Our previous work applies GLS-GAN [15] and DST [16] to generate simulation images. For each defect type, GLS-GAN can transplant existing defects into defect-free samples through regional paired iteration to generate high-quality defect images. But it still needs dozens of reference samples to reduce the similarity, and takes a long time to generate enough images. DST can generate different defect images with only one single reference. However, it cannot accurately generate the texture structure of the defect area.
In view of the above, we decide to investigate a tougher task: how to efficiently train the defect segmentation network using only one single defect image for each type of defect. There are two main difficulties: how to ensure the diversity of augmented training samples to avoid over-fitting, and how to ensure the effectivity of image augmentation with a complex background, which guarantees that the segmentation network can efficiently converge when trained with the augmented samples.
Based on the former work, we propose a novel Defect Segmentation with Simulation (DSS) framework. We depart the whole task into two parts, which are defect sample simulation and defect segmentation, and propose a two-stage defect simulation algorithm and a residual-restored-based segmentation algorithm respectively. In the simulation framework, we first use a regular GAN [17] structure for coarsely matching. Once trained with regional unpaired training strategy, it can conduct forward generation of corresponding defect types in areas of any specified shape. And then, we take the style transfer in DST as the second stage to merge the generated area into the defect-free images. The advantage of the two-stage algorithm is that it can disassemble the tough task of defect simulation upon a complex background texture, allowing each stage to focus on completing a single specific and easier function. Samples of the simulation algorithm are shown in Fig. 1. The paired simulated defect images along with their corresponding defect-free samples are then used to train the lightweight residual-restored-based segmentation network, which can restore the defect and further yield the prediction through the residual image. The backbones of the three networks share the same encoder-decoder structure which is easy to implement. And we are inspired by MobileNet [18] to apply depthwise separable convolution on the segmentation algorithm. Finally, we verify the proposed method in button [15], road crack [19], silicon steel strip [20], [21] datasets. In the case of using only defect-free samples and one single defect image for each type, the proposed method outperforms related algorithms in recent years. And the segmentation algorithm can achieve convergence within 30 epochs on all the three datasets. We also design an ablation experiment to further prove the improvement effect of each module.
In summary, our contributions are:
- 1.
We investigate a new task of training a defect segmentation algorithm with single defect reference and defect-free samples, and propose a novel Defect Segmentation with Simulation (DSS) framework to solve it.
- 2.
Based on our previous simulation work, we adapt regional unpaired training strategy on a regular GAN, to realize the forward generation of defects within any specified shape. In the meanwhile, we apply the style transfer in DST to blend the generated area into the defect-free samples.
- 3.
We design a novel lightweight residual-restored-based segmentation algorithm. And we use paired training strategy with three optimizers for further improvement of segmentation accuracy.
- 4.
We adopt a simple backbone network in the framework as well as apply depthwise separable convolution and multiple losses to the segmentation task for more efficient training.
Section snippets
Dataset augmentation
In the classification task, adopting translation with rotation [8], random cropping and mirroring [12], adding random noise [9] or other methods for data augmentation can achieve better results. Besides, some researches, such as [10], merge the images from the same class to generate new samples.
In the target detection task, it is necessary to consider the label image while augmenting the dataset, and the method of random clipping [11] is often used to improve the accuracy of the model.
Methods
This chapter will detail our proposed method. For the task of defect segmentation with only one single reference sample for each defect type, we design a simulation algorithm to obtain the simulated defect images from defect-free images as dataset augmentation. In addition, we design a defect segmentation algorithm that can detect various kinds of real defects after trained with the simulated images. The framework of the entire method is shown in Fig. 2.
There are two major
Conclusion
In this paper, we propose a few-sample surface defect segmentation method based on defect sample simulation to deal with the scarcity of the real defect training samples. In order to generate various simulated samples with high quality, we use an adversarial network to reconstruct the structure of defect in the specified area of the defect-free samples, and then adopt the neural style transfer to blend the simulated defect into the background. Further, in order to make full use of the paired
CRediT authorship contribution statement
Taoran Wei: Conceptualization, Methodology, Investigation, Software, Formal analysis, Visualization, Writing - original draft. Danhua Cao: Resources, Project administration, Supervision, Writing - review & editing. Caiyun Zheng: Investigation, Validation, Formal analysis, Visualization, Writing - review & editing. Qun Yang: Methodology, Writing - review & editing, Data curation.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgement
We gratefully acknowledge the support from Department of science and technology, Hubei Provincial People’s Government. This research is a part of 2019AAA057 project.
Taoran Wei received the bachelor’s degree in School of Optical and Electronic Information from Huazhong Uni- versity of Science and Technology. He is currently pursu- ing a Ph.D. in Huazhong University of Science and Tech- nology. His main research areas are machine vision and quality inspection.
References (55)
- et al.
Wavelet based methods on patterned fabric defect detection
Pattern Recognit.
(2005) - et al.
Automated surface inspection for statistical textures
Image Vis. Comput.
(2003) - et al.
Automatic fabric defect detection with a wide-and-compact network
Neurocomputing
(2019) - et al.
Anomaly detection with convolutional neural networks for industrial surface inspection
Procedia CIRP
(2019) - et al.
Defect detection of hot rolled steels with a new object detection framework called classification priority network
Comput. Ind. Eng.
(2019) - et al.
Defective samples simulation through adversarial training for automatic surface inspection
Neurocomputing
(2019) - et al.
A novel approach for defect detection on vessel structures using saliency-related features
Ocean Eng.
(2018) Automated defect detection in textured surfaces using optimal elliptical Gabor filters
Optik
(2015)- et al.
Fabric defect inspection based on lattice segmentation and Gabor filtering
Neurocomputing
(2017) - et al.
Accurate defect detection via sparsity reconstruction for weld radiographs
NDT E Int.
(2018)
Tiny surface defect inspection of electronic passive components using discrete cosine transform decomposition and cumulative sum techniques
Image Vis. Comput.
Fabric Defect Detection Using Auto-Correlation Function
Int. J. Comput. Theory Eng.
A Generic Deep-Learning-Based Approach for Automated Surface Inspection
IEEE Trans. Cybern.
Smart Augmentation Learning an Optimal Data Augmentation Strategy
IEEE Access
A Surface Defect Detection Method Based on Positive Samples
An Unsupervised-Learning-Based Approach for Automated Defect Inspection on Textured Surfaces
IEEE Trans. Instrum. Meas.
Automatic crack detection on 2D pavement images: An algorithm based on minimal path selection
IEEE Trans. Intell. Transp. Syst.
Surface Defect Detection Method Using Saliency Linear Scanning Morphology for Silicon Steel Strip under Oil Pollution Interference
ISIJ Int.
Micro Surface Defect Detection Method for Silicon Steel Strip Based on Saliency Convex Active Contour Model
Math. Probl. Eng.
Cited by (9)
Automatic surface defect segmentation for hot-rolled steel strip using depth-wise separable U-shape network
2021, Materials LettersCitation Excerpt :For example, Song et al. adopted an encoder-decoder residual network for defect segmentation [6]. Wei et al., proposed a generation adversarial net and neural style transfer to segment defects for silcon steel strips [7]. With the expensive labeling cost, the research for hot-rolled steel strip defect segmentation is limited.
A steel surface defect detection model based on graph neural networks
2024, Measurement Science and TechnologyDeep Industrial Image Anomaly Detection: A Survey
2024, Machine Intelligence ResearchKnowledge-Driven Metal Coating Defect Segmentation
2024, Dianzi Keji Daxue Xuebao/Journal of the University of Electronic Science and Technology of ChinaReview of surface defect detection of steel products based on machine vision
2023, IET Image Processing
Taoran Wei received the bachelor’s degree in School of Optical and Electronic Information from Huazhong Uni- versity of Science and Technology. He is currently pursu- ing a Ph.D. in Huazhong University of Science and Tech- nology. His main research areas are machine vision and quality inspection.
Danhua Cao is a professor in the School of Optical and Electronic Information, Huazhong University of Science and Technology. She received her Ph.D. degree in elec- tronic physics and devices from Huazhong University of Science and Technology in 1993. She is a permanent member of the Professional Committee of Opto-electronic Technology in the Chinese Optical Society. Her research interests include optoelectronic sensing and signal pro- cessing as well as machine vision algorithms and systems
Caiyun Zheng received the bachelor's degree in School of Optical and Electronic Information from Huazhong University of Science and Technology. She is currently studying for a master's degree at the Huazhong University of Science and Technology. Her research interests include machine vision, deep learning and automated surface inspection
Qun Yang received his B.E. degree in 2017 from the School of Optical and Electronic Information, Huazhong University of Science and Technology, where he is currently pursuing his M.E. degree. He is interested in the field of image processing, machine learning.