Skip to main content

Advertisement

SpringerLink
Log in

Explainable AI for domain experts: a post Hoc analysis of deep learning for defect classification of TFT–LCD panels

  • Published:
Journal of Intelligent Manufacturing Aims and scope Submit manuscript

Abstract

The deep learning (DL) model has performed successfully in various fields, including manufacturing. DL models for defect image data analysis in the manufacturing field have been applied to multiple domains such as defect detection, classification, and localization. However, DL models require trade-offs in accuracy and interpretability. We use explainable artificial intelligence techniques to analyze the predicted results of the defect image classification model, which is considered as a “black-box” model, to produce human-understandable results. We visualize defects using layer-wise relevance propagation-based methods, fit the model into a decision tree, and convert prediction results into human-interpretable text. Our research complements the interpretation of prediction results for the classification model. The domain expert can obtain the reliability and explanatory ability for the defect classification of TFT–LCD panel data of the DL model through the results of the proposed analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  • Adadi, A., & Berrada, M. (2018). Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI). IEEE Access, 6, 52138–52160.

    Article  Google Scholar 

  • Alber, M., Lapuschkin, S., Seegerer, P., Hägele, M., Schütt, K. T., Montavon, G., et al. (2019). iNNvestigate Neural Networks. Journal of Machine Learning Research, 20(93), 1–8.

    Google Scholar 

  • Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., Chatila, R., & Herrera, F. (2019). Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI. arXiv preprint arXiv:1910.10045.

  • Asif, K., Zhang, L., Derrible, S., Indacochea, J. E., Ozevin, D., & Ziebart, B. (2020). Machine learning model to predict welding quality using air-coupled acoustic emission and weld inputs. Journal of Intelligent Manufacturing pp. 1–15.

  • Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K. R., & Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7), e0130140.

    Article  Google Scholar 

  • Badmos, O., Kopp, A., Bernthaler, T., & Schneider, G. (2020). Image-based defect detection in lithium-ion battery electrode using convolutional neural networks. Journal of Intelligent Manufacturing, 31(4), 885–897.

    Article  Google Scholar 

  • Bae, J., Kim, M., & Lim, J. S. (2020). Feature Extraction Model Based on Inception V3 to Distinguish Normal Heart Sound from Systolic Murmur. In: 2020 International Conference on Information and Communication Technology Convergence (ICTC), pp. 460–463. IEEE.

  • Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M. F., & Eckersley, P. (2019). Explainable Machine Learning in Deployment. arXiv preprint arXiv:1909.06342 pp. 648–657.

  • Buhrmester, V., Münch, D., & Arens, M. (2019). Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey. arXiv preprint arXiv:1911.12116 pp. 1–22.

  • Calderisi, M., Galatolo, G., Ceppa, I., Motta, T., & Vergentini, F. (2019). Improve Image Classification Tasks Using Simple Convolutional Architectures with Processed Metadata Injection. In: 2019 IEEE Second International Conference on Artificial Intelligence and Knowledge Engineering (AIKE), pp. 223–230. IEEE.

  • Derigent, W., Cardin, O., & Trentesaux, D. (2020). Industry 4.0: contributions of holonic manufacturing control architectures and future challenges. Journal of Intelligent Manufacturing pp. 1–22.

  • Di, H., Ke, X., Peng, Z., & Dongdong, Z. (2019). Surface defect classification of steels with a new semi-supervised learning method. Optics and Lasers in Engineering 117(December 2018), 40–48.

  • Dionson, M. G. D., & El Jireh, P. B. (2020). Inception-V3 Architecture in Dermatoglyphics-Based Temperament Classification. Philippine Social Science Journal, 3(2), 173–174.

    Article  Google Scholar 

  • Dobrescu, A., Valerio Giuffrida, M., & Tsaftaris, S. A. (2019). Understanding deep neural networks for regression in leaf counting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.

  • Du, W., Shen, H., Fu, J., Zhang, G., Shi, X., & He, Q. (2021). Automated detection of defects with low semantic information in X-ray images based on deep learning. Journal of Intelligent Manufacturing, 32(1), 141–156.

    Article  Google Scholar 

  • Ferretti, E., Tamargo, L. H., García, A. J., Errecalde, M. L., & Simari, G. R. (2017). An approach to decision making based on dynamic argumentation systems. Artificial Intelligence, 242, 107–131.

    Article  Google Scholar 

  • Ghai, B., Liao, Q. V., Zhang, Y., Bellamy, R., & Mueller, K. (2020). Explainable Active Learning (XAL): An Empirical Study of How Local Explanations Impact Annotator Experience. arXiv preprint arXiv:2001.09219.

  • Ghotra, B., McIntosh, S., & Hassan, A. E. (2017). A large-scale study of the impact of feature selection techniques on defect classification models. In: 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), pp. 146–157. IEEE.

  • Gray, N. A. (1984). Applications of artificial intelligence for organic chemistry: Analysis of C-13 spectra. Artificial Intelligence, 22(1), 1–21.

    Article  Google Scholar 

  • Grezmak, J., Zhang, J., Wang, P., Loparo, K. A., & Gao, R. X. (2019). Interpretable Convolutional Neural Network Through Layer-wise Relevance Propagation for Machine Fault Diagnosis. IEEE Sensors Journal, 20(6), 3172–3181.

    Article  Google Scholar 

  • Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2018). A survey of methods for explaining black box models. ACM Computing Surveys, 51(5), 1–45.

    Article  Google Scholar 

  • Gunning, D. (2017). Explainable artificial intelligence (XAI). Defense Advanced Research Projects Agency (DARPA), nd Web 2.

  • Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2018). Metrics for explainable AI: Challenges and prospects. arXiv preprint arXiv:1812.04608.

  • Hsieh, Y. C., Chin, C. L., Wei, C. S., Chen, I. M., Yeh, P. Y., & Tseng, R. J. (2020). Combining VGG16, Mask R-CNN and Inception V3 to identify the benign and malignant of breast microcalcification clusters. In: 2020 International Conference on Fuzzy Theory and Its Applications (iFUZZY), pp. 1–4. IEEE.

  • Jin, C. H., Kim, H. J., Piao, Y., Li, M., & Piao, M. (2020). Wafer map defect pattern classification based on convolutional neural network features and error-correcting output codes. Journal of Intelligent Manufacturing, 31(8), 1–15.

    Article  Google Scholar 

  • Kim, B. C., Yoon, J. S., Choi, J. S., & Suk, H. I. (2019). Multi-scale gradual integration CNN for false positive reduction in pulmonary nodule detection. Neural Networks, 115, 1–10.

    Article  Google Scholar 

  • Kim, M., Lee, M., An, M., & Lee, H. (2019). Effective automatic defect classification process based on CNN with stacking ensemble model for TFT-LCD panel. Journal of Intelligent Manufacturing, 31(5), 1165–1174.

    Article  Google Scholar 

  • Kim, S. T., Lee, H., Kim, H. G., & Ro, Y. M. (2018). Icadx: Interpretable computer aided diagnosis of breast masses. In: Medical Imaging 2018: Computer-Aided Diagnosis, vol. 10575, p. 1057522. International Society for Optics and Photonics.

  • Konstantakopoulos, I. C., Barkan, A. R., He, S., Veeravalli, T., Liu, H., & Spanos, C. (2019). A deep learning and gamification approach to improving human-building interaction and energy efficiency in smart infrastructure. Applied Energy, 237, 810–821.

    Article  Google Scholar 

  • Kumar, A. (2008). Computer-vision-based fabric defect detection: A survey. IEEE Transactions on Industrial Electronics, 55(1), 348–363.

    Article  Google Scholar 

  • Kuwajima, H., Tanaka, M., & Okutomi, M. (2019). Improving transparency of deep neural inference process. Progress in Artificial Intelligence, 8(2), 273–285.

    Article  Google Scholar 

  • Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K. R. (2019). Unmasking clever hans predictors and assessing what machines really learn. Nature Communications, 10(1), 1–8.

    Article  Google Scholar 

  • Lee, J. H., & Lee, J. H. (2019). A Reliable Defect Detection Method for Patterned Wafer Image Using Convolutional Neural Networks with the Transfer Learning. In: IOP Conference Series: Materials Science and Engineering, vol. 647, p. 012010. IOP Publishing.

  • Lee, S. Y., Tama, B. A., Moon, S. J., & Lee, S. (2019). Steel surface defect diagnostics using deep convolutional neural network and class activation map. Applied Sciences (Switzerland), 9(24), 5449.

    Article  Google Scholar 

  • Leonetti, M., Iocchi, L., & Stone, P. (2016). A synthesis of automated planning and reinforcement learning for efficient, robust decision-making. Artificial Intelligence, 241, 103–130.

    Article  Google Scholar 

  • Li, J., & Fang, P. (2019). FVGNN: A Novel GNN to Finger Vein Recognition from Limited Training Data. In: 2019 IEEE 8th Joint International Information Technology and Artificial Intelligence Conference (ITAIC), pp. 144–148. IEEE.

  • Li, X., Jia, X., Yang, Q., & Lee, J. (2020). Quality analysis in metal additive manufacturing with deep learning. Journal of Intelligent Manufacturing, 31(8), 2003–2017.

    Article  Google Scholar 

  • Li, X., Zhang, W., & Ding, Q. (2019). Understanding and improving deep learning-based rolling bearing fault diagnosis with attention mechanism. Signal Processing, 161, 136–154.

    Article  Google Scholar 

  • Liu, C., Tan, J., & Wang, X. (2020). A data-driven decision-making optimization approach for inconsistent lithium-ion cell screening. Journal of Intelligent Manufacturing, 31(4), 833–845.

    Article  Google Scholar 

  • Marcos, D., Lobry, S., & Tuia, D. (2019). Semantically Interpretable Activation Maps: what-where-how explanations within CNNs. arXiv preprint arXiv:1909.08442.

  • Ming, Y. (2017). A Survey on Visualization for Explainable Classifiers.

  • Mohseni, S., Zarei, N., & Ragan, E. D. (2018). A survey of evaluation methods and measures for interpretable machine learning. arXiv preprint arXiv:1811.11839.

  • Montavon, G., Binder, A., Lapuschkin, S., Samek, W., & Müller, K. R. (2019). Layer-Wise Relevance Propagation: An Overview. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 11700 LNCS, 193–209.

  • Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K. R. (2017) Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recognition 65(November 2016), 211–222.

  • Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K. R. (2017). Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognition, 65, 211–222.

    Article  Google Scholar 

  • Morales, D., Talavera, E., & Remeseiro, B. (2020). Playing to distraction: towards a robust training of CNN classifiers through visual explanation techniques. arXiv preprint arXiv:2012.14173.

  • Ning, F., Shi, Y., Cai, M., & Xu, W. (2020). Various realization methods of machine-part classification based on deep learning. Journal of Intelligent Manufacturing., 31(8), 1–14.

    Article  Google Scholar 

  • Peng, Y., Dharssi, S., Chen, Q., Keenan, T. D., Agrón, E., Wong, W. T., et al. (2019). DeepSeeNet: A Deep Learning Model for Automated Classification of Patient-based Age-related Macular Degeneration Severity from Color Fundus Photographs. Ophthalmology, 126(4), 565–575.

    Article  Google Scholar 

  • Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “Why should I trust you?” Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135–1144.

  • Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on Image Data Augmentation for Deep Learning. Journal of Big Data, 6(1).

  • Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.

  • Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017). Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825.

  • Soumik, M. F. I., & Hossain, M. A. (2020). Brain Tumor Classification With Inception Network Based Deep Learning Model Using Transfer Learning. In: 2020 IEEE Region 10 Symposium (TENSYMP), pp. 1018–1021. IEEE.

  • Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2015). Striving for simplicity: The all convolutional net. 3rd International Conference on Learning Representations, ICLR 2015 - Workshop Track Proceedings pp. 1–14.

  • Sturm, I., Lapuschkin, S., Samek, W., & Müller, K. R. (2016). Interpretable deep neural networks for single-trial EEG classification. Journal of Neuroscience Methods, 274, 141–145.

    Article  Google Scholar 

  • Sundararajan, M., Taly, A., & Yan, Q. (2016). Gradients of counterfactuals. arXiv preprint arXiv:1611.02639.

  • Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9.

  • Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818–2826.

  • Wang, J., Zhu, X., & Gong, S. (2017). Discovering visual concept structure with sparse and incomplete tags. Artificial Intelligence, 250, 16–36.

    Article  Google Scholar 

  • Wang, P., Ananya, Yan, R., & Gao, R. X. (2017). Virtualization and deep recognition for system fault classification. Journal of Manufacturing Systems 44(November), 310–316.

  • Xia, C., Pan, Z., Polden, J., Li, H., Xu, Y., & Chen, S. (2021). Modelling and prediction of surface roughness in wire arc additive manufacturing using machine learning. Journal of Intelligent Manufacturing pp. 1–16.

  • Xia, X., Xu, C., & Nan, B. (2017). Inception-v3 for flower classification. In: 2017 2nd International Conference on Image, Vision and Computing (ICIVC), pp. 783–787. IEEE.

  • Xie, M., Jean, N., Burke, M., Lobell, D., & Ermon, S. (2016). Transfer learning from deep features for remote sensing and poverty mapping. 30th AAAI Conference on Artificial Intelligence, AAAI 2016 pp. 3929–3935.

  • Xu, X., Zheng, H., Guo, Z., Wu, X., & Zheng, Z. (2019). SDD-CNN: Small data-driven convolution neural networks for subtle roller defect inspection. Applied Sciences, 9(7), 1364.

    Article  Google Scholar 

  • Zeiler, M. D., & Fergus, R. (2014). Visualizing and understanding convolutional networks. In: European conference on computer vision, pp. 818–833. Springer.

  • Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016). Learning Deep Features for Discriminative Localization. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Download references

Acknowledgements

The authors appreciate that the aim systems, Inc. provided the data for the study.

Author information

Authors and Affiliations

  1. Department of Industrial Management Engineering, Korea University, Seoul, Korea

    Minyoung Lee, Joohyoung Jeon & Hongchul Lee

Authors
  1. Minyoung Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joohyoung Jeon
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hongchul Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongchul Lee.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lee, M., Jeon, J. & Lee, H. Explainable AI for domain experts: a post Hoc analysis of deep learning for defect classification of TFT–LCD panels. J Intell Manuf 33, 1747–1759 (2022). https://doi.org/10.1007/s10845-021-01758-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10845-021-01758-3

Keywords

Search

Navigation