Skip to main content
SpringerLink
Log in

Automated visual detection of geometrical defects in composite manufacturing processes using deep convolutional neural networks

  • Published:
Journal of Intelligent Manufacturing Aims and scope Submit manuscript

Abstract

Detection of fiber composite material boundaries and defects is critical to the automation of the manufacturing process in the aviation industry. This paper describes a process to evaluate four well-performing deep convolutional neural network models (Mask R-CNN, U-Net, DeepLab V3+, and IC-Net) for use in such a process. A custom-captured dataset of images showing fiber cut-pieces with geometrical defects was annotated and augmented for training deep convolutional neural network models; results show acceptable detection accuracy for gripper and fabric based on the Intersection over Union (IoU) scores of up to 0.92 and 0.86, respectively. However, wrinkle detection initially achieves a significantly lower IoU score of 0.40 in the best case. This discrepancy is mainly due to geometrical ambiguities, as wrinkles do not have a clearly defined boundary and are hard to distinguish even for human eye. The model is then evaluated as a binary predictor based on per-component detection success; the model achieves a recall rate (i.e., the ratio of the wrinkles detected to all existing wrinkles) of 0.71 and a precision score (i.e., the ratio of those detected being actually wrinkles) of 0.76. From a practical point of view, this model can outperform a human operator based on the results presented. Two complementary approaches are also introduced for the detection of wrinkles at the early stages of formation as well as the completely formed wrinkles. The developed method can be readily used in a variety of composite manufacturing processes or adapted to other similar tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. Intersection over Union.

References

  • Abdul Ghafour, T., Colmars, J., & Boisse, P. (2019). The importance of taking into account behavior irreversibilities when simulating the forming of textile composite reinforcements. Composites Part A: Applied Science and Manufacturing. https://doi.org/10.1016/j.compositesa.2019.105641

    Article  Google Scholar 

  • Alshahrani, H., & Hojjati, M. (2017). A new test method for the characterization of the bending behavior of textile prepregs. Composites Part A: Applied Science and Manufacturing, 97(97), 128–140. https://doi.org/10.1016/j.compositesa.2017.02.027

    Article  Google Scholar 

  • Anantharaman, R., Velazquez, M., & Lee, Y. (2018). Utilizing mask R-CNN for detection and segmentation of oral diseases. In 2018 IEEE international conference on bioinformatics and biomedicine (BIBM) (pp. 2197–2204).

  • Aridhi, A., Arfaoui, M., Mabrouki, T., Naouar, N., Denis, Y., Zarroug, M., & Boisse, P. (2019). Textile composite structural analysis taking into account the forming process. Composites Part B: Engineering, 166, 773–784. https://doi.org/10.1016/j.compositesb.2019.02.047

    Article  Google Scholar 

  • Bian, X., Lim, S. N., & Zhou, N. (2016). Multiscale fully convolutional network with application to industrial inspection. In 2016 IEEE winter conference on applications of computer vision, WACV 2016. https://doi.org/10.1109/WACV.2016.7477595

  • Boisse, P., Colmars, J., Hamila, N., Naouar, N., & Steer, Q. (2018). Bending and wrinkling of composite fiber preforms and prepregs. A review and new developments in the draping simulations. Composites Part B: Engineering. https://doi.org/10.1016/j.compositesb.2017.12.061

    Article  Google Scholar 

  • Cao, J., Akkerman, R., Boisse, P., Chen, J., Cheng, H. S., De Graaf, E. F., Gorczyca, J. L., Harrison, P., Hivet, G., Launay, J., & Lee, W. (2008). Characterization of mechanical behavior of woven fabrics: Experimental methods and benchmark results. Composites Part A: Applied Science and Manufacturing, 39(6), 1037–1053. https://doi.org/10.1016/j.compositesa.2008.02.016

    Article  Google Scholar 

  • Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2017a). Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4), 834–848

    Article  Google Scholar 

  • Chen, L.-C., Papandreou, G., Schroff, F., & Adam, H. (2017). Rethinking atrous convolution for semantic image segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition.

  • Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018). Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV) (pp. 801–818).

  • Dangora, L. M., Mitchell, C., White, K. D., Sherwood, J. A., & Parker, J. C. (2018). Characterization of temperature-dependent tensile and flexural rigidities of a cross-ply thermoplastic lamina with implementation into a forming model. International Journal of Material Forming, 11(1), 43–52. https://doi.org/10.1007/s12289-016-1327-2

    Article  Google Scholar 

  • Dempster, A. P. (2008). Upper and lower probabilities induced by a multivalued mapping. In Classic works of the dempster-shafer theory of belief functions (pp. 57–72). Springer.

  • Denis, Y., Morestin, F., & Hamila, N. (2020). A hysteretic model for fiber-reinforced composites at finite strains: Fractional derivatives, computational aspects and analysis. Computational Materials Science, 181, 109716. https://doi.org/10.1016/j.commatsci.2020.109716

    Article  Google Scholar 

  • Djavadifar, A., Graham-Knight, J. B., Gupta, K., Körber, M., Lasserre, P., & Najjaran, H. (2019). Robot-assisted composite manufacturing based on machine learning applied to multi-view computer vision. In International conference on smart multimedia.

  • Doitrand, A., Fagiano, C., Irisarri, F. X., & Hirsekorn, M. (2015). Comparison between voxel and consistent meso-scale models of woven composites. Composites Part A: Applied Science and Manufacturing, 73(73), 143–154. https://doi.org/10.1016/j.compositesa.2015.02.022

    Article  Google Scholar 

  • Dong, X., Taylor, C. J., & Cootes, T. F. (2019). Small defect detection using convolutional neural network features and random forests. In Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), 11132 LNCS (pp. 398–412). https://doi.org/10.1007/978-3-030-11018-5_35

  • Enshaei, N., Ahmad, S., & Naderkhani, F. (2020). Automated detection of textured-surface defects using UNet-based semantic segmentation network. In Proceedings of the annual conference of the prognostics and health management society, PHM, 2020-June. https://doi.org/10.1109/ICPHM49022.2020.9187023

  • Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010). The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2), 303–338. https://doi.org/10.1007/s11263-009-0275-4

    Article  Google Scholar 

  • Faal, R. T., Sourki, R., Crawford, B., Vaziri, R., & Milani, A. S. (2020). Using fractional derivatives for improved viscoelastic modeling of textile composites. Part II: Fabric under different temperatures. Composite Structures, 248, 112494. https://doi.org/10.1016/j.compstruct.2020.112494

    Article  Google Scholar 

  • Ferguson, M., Ak, R., Lee, Y. T. T., & Law, K. H. (2018). Detection and segmentation of manufacturing defects with convolutional neural networks and transfer learning. Smart and Sustainable Manufacturing Systems, 2(1), 137–164. https://doi.org/10.1520/SSMS20180033

    Article  Google Scholar 

  • Girshick, R. (2015). Fast R-CNN. In Proceedings of the IEEE international conference on computer vision, 2015 Inter (pp. 1440–1448). https://doi.org/10.1109/ICCV.2015.169

  • Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR.2014.81

    Article  Google Scholar 

  • Gupta, K., Körber, M., Djavadifar, A., Krebs, F., & Najjaran, H. (2019). Wrinkle and boundary detection of fiber products in robotic composites manufacturing. Assembly Automation, 40(2), 283–291. https://doi.org/10.1108/AA-05-2019-0094

    Article  Google Scholar 

  • Gupta, K, Körber, M., Krebs, F., & Najjaran, H. (2018). Vision-based deformation and wrinkle detection for semi-finished fiber products on curved surfaces. In 2018 IEEE 14th international conference on automation science and engineering (CASE) (pp. 618–623). https://doi.org/10.1109/COASE.2018.8560559

  • Habboush, A., Sanbhal, N., Shao, H., Jiang, J., & Chen, N. (2018). Characterization and analysis of in-plane shear behavior of glass warp-knitted non-crimp fabrics based on picture frame method. Materials. https://doi.org/10.3390/ma11091550

    Article  Google Scholar 

  • He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017). Mask R-CNN. In Proceedings of the IEEE international conference on computer vision, 2017-Octob (pp. 2980–2988). https://doi.org/10.1109/ICCV.2017.322

  • Huang, H., Wei, Z., & Yao, L. (2019). A novel approach to component assembly inspection based on mask R-CNN and support vector machines. Information (Switzerland). https://doi.org/10.3390/info10090282

    Article  Google Scholar 

  • Huang, J., Boisse, P., Hamila, N., & Zhu, Y. (2020). Simulation of wrinkling during bending of composite reinforcement laminates. Materials, 13(10), 2374. https://doi.org/10.3390/ma13102374

    Article  Google Scholar 

  • Iwata, A., Inoue, T., Naouar, N., Boisse, P., & Lomov, S. V. (2018). Meso-macro simulation of the woven fabric local deformation in draping. In AIP conference proceedings (Vol. 1960, p. 020012). American Institute of Physics Inc. https://doi.org/10.1063/1.5034813

  • Karim, M. M., Doell, D., Lingard, R., Yin, Z., Leu, M. C., & Qin, R. (2019). A region-based deep learning algorithm for detecting and tracking objects in manufacturing plants. Procedia Manufacturing, 39, 168–177. https://doi.org/10.1016/j.promfg.2020.01.289

    Article  Google Scholar 

  • Körber, M., Manufacturing, C. F.-P., & 2019, undefined. (n.d.). Automated planning and optimization of a draping processes within the CATIA environment using a Python Software Tool. Elsevier. https://www.sciencedirect.com/science/article/pii/S2351978920301141. Accessed February 26, 2020.

  • Liang, B., Colmars, J., & Boisse, P. (2017). A shell formulation for fibrous reinforcement forming simulations. Composites Part A: Applied Science and Manufacturing, 100, 81–96. https://doi.org/10.1016/j.compositesa.2017.04.024

    Article  Google Scholar 

  • Liang, B., Colmars, J., & Boisse, P. (2018). A shell approach for fibrous reinforcement forming simulations. In AIP conference proceedings (Vol. 1960, p. 020015). American Institute of Physics Inc. https://doi.org/10.1063/1.5034816

  • Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., et al. (2014). Microsoft coco: Common objects in context. In European conference on computer vision (pp. 740–755).

  • Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3431–3440).

  • Mehdikhani, M., Petrov, N., Gorbatikh, L., & Lomov, S. V. (2018). Multi-scale experimental and computational investigation of matrix cracking evolution in carbon fiber-reinforced composites in the absence and presence of voids. In IOP conference series: Materials science and engineering (Vol. 406, p. 012011). Institute of Physics Publishing. https://doi.org/10.1088/1757-899X/406/1/012011

  • Montazerian, H., Sourki, R., Ramezankhani, M., Rashidi, A., Koerber, M., & Milani, A. S. (2019). Digital twining of an automated fabric draping process for industry 4.0 applications: Part imulti-body simulation and finite element modeling. In CAMX 2019: composites and advanced materials expo. https://elib.dlr.de/131180/. Accessed February 25, 2020.

  • Peng, X. Q., Cao, J., Chen, J., Xue, P., Lussier, D. S., & Liu, L. (2004). Experimental and numerical analysis on normalization of picture frame tests for composite materials. Composites Science and Technology, 64(1), 11–21. https://doi.org/10.1016/S0266-3538(03)00202-1

    Article  Google Scholar 

  • Poppe, C., Rosenkranz, T., Dörr, D., & Kärger, L. (2019). Comparative experimental and numerical analysis of bending behaviour of dry and low viscous infiltrated woven fabrics. Composites Part A: Applied Science and Manufacturing. https://doi.org/10.1016/j.compositesa.2019.05.034

    Article  Google Scholar 

  • Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. In Proceedings of the IEEE computer society conference on computer vision and pattern recognition (Vol. 2016-Decem, pp. 779–788). https://doi.org/10.1109/CVPR.2016.91

  • Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems (pp. 91–99).

  • Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics) (Vol. 9351, pp. 234–241). Springer Verlag. https://doi.org/10.1007/978-3-319-24574-4_28

  • Singh, J., & Shekhar, S. (2018). Road damage detection and classification in smartphone captured images using mask R-CNN. In IEEE international conference on big data cup (Vol. abs/1811.0).

  • Tabernik, D., Šela, S., Skvarč, J., & Skočaj, D. (2020). Segmentation-based deep-learning approach for surface-defect detection. Journal of Intelligent Manufacturing, 31(3), 759–776. https://doi.org/10.1007/s10845-019-01476-x

    Article  Google Scholar 

  • Tang, T. W., Kuo, W. H., Lan, J. H., Ding, C. F., Hsu, H., & Young, H. T. (2020). Anomaly detection neural network with dual auto-encoders GAN and its industrial inspection applications. Sensors (Switzerland). https://doi.org/10.3390/s20123336

    Article  Google Scholar 

  • Voggenreiter, H., & Nieberl, D. (2015). AZIMUT Abschlussbericht. TIB.

  • Wang, D., Naouar, N., Vidal-Salle, E., & Boisse, P. (2018). Longitudinal compression and Poisson ratio of fiber yarns in meso-scale finite element modeling of composite reinforcements. Composites Part B: Engineering, 141, 9–19. https://doi.org/10.1016/j.compositesb.2017.12.042

    Article  Google Scholar 

  • Wang, K. J., Rizqi, D. A., & Nguyen, H. P. (2020). Skill transfer support model based on deep learning. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-020-01606-w

    Article  Google Scholar 

  • Wei, W., Zhang, C., & Deng, D. (2020). Content estimation of foreign fibers in cotton based on deep learning. Electronics (Switzerland), 9(11), 1–22. https://doi.org/10.3390/electronics9111795

    Article  Google Scholar 

  • Wen, V., Wong, H., Ferguson, M., Law, K. H., Lee, Y.-T. T., & Witherell, P. (2020). Automatic volumetric segmentation of additive manufacturing defects with 3D U-Net. SSS20 (VIRTUAL) AAAI spring symposium on AI in manufacturing. https://www.nist.gov/publications/automatic-volumetric-segmentation-additive-manufacturing-defects-3d-u-net

  • Xue, P., Peng, X., & Cao, J. (2003). A non-orthogonal constitutive model for characterizing woven composites. Composites Part A: Applied Science and Manufacturing, 34(2), 183–193. https://doi.org/10.1016/S1359-835X(02)00052-0

    Article  Google Scholar 

  • Yan, S., Zeng, X., & Long, A. (2019). Meso-scale modelling of 3D woven composite T-joints with weave variations. Composites Science and Technology, 171, 171–179. https://doi.org/10.1016/j.compscitech.2018.12.024

    Article  Google Scholar 

  • Zhao, G., Hu, J., Xiao, W., & Zou, J. (2020). A mask R-CNN based method for inspecting cable brackets in aircraft. Chinese Journal of Aeronautics. https://doi.org/10.1016/j.cja.2020.09.024

    Article  Google Scholar 

  • Zhao, H., Qi, X., Shen, X., Shi, J., & Jia, J. (2018). ICNet for real-time semantic segmentation on high-resolution images. In Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), 11207 LNCS (pp. 418–434). https://doi.org/10.1007/978-3-030-01219-9_25

Download references

Acknowledgements

The authors would like to acknowledge the support of the Natural Sciences and Engineering Council Canada (NSERC) and Kinova Robotics under the NSERC Collaborative Research and Development (CRD) program Grant CRDPJ 543881-19, and the German Aerospace Center (DLR) towards this research.

Author information

Authors and Affiliations

  1. School of Engineering, University of British Columbia, 3333 University Way, Kelowna, BC, V1V 1V7, Canada

    Abtin Djavadifar & Homayoun Najjaran

  2. Department of Computer Science, University of British Columbia, 3333 University Way, Kelowna, BC, V1V 1V7, Canada

    John Brandon Graham-Knight & Patricia Lasserre

  3. Center for Lightweight-Production-Technology, German Aerospace Center, Am Technologiezentrum 4, 86159, Augsburg, Germany

    Marian Kӧrber

Authors
  1. Abtin Djavadifar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. John Brandon Graham-Knight
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marian Kӧrber
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Patricia Lasserre
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Homayoun Najjaran
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Homayoun Najjaran.

Additional information

Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Djavadifar, A., Graham-Knight, J.B., Kӧrber, M. et al. Automated visual detection of geometrical defects in composite manufacturing processes using deep convolutional neural networks. J Intell Manuf 33, 2257–2275 (2022). https://doi.org/10.1007/s10845-021-01776-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10845-021-01776-1

Keywords

Search

Navigation