Hierarchical Deep Learning Neural Network (HiDeNN): An artificial intelligence (AI) framework for computational science and engineering

https://doi.org/10.1016/j.cma.2020.113452Get rights and content

Highlights

  • An AI system framework for generally challenging problems in computational science and engineering.

  • Demonstrating how to build HiDeNN from hierarchically assembled deep neural networks.

  • Application of HiDeNN to capture stress concentration and solve multiscale problem.

  • Discovery of non-dimensional number with HiDeNN based on experimental data.

  • Vision on how to deploy HiDeNN for three examples of challenging problems.

Abstract

In this work, a unified AI-framework named Hierarchical Deep Learning Neural Network (HiDeNN) is proposed to solve challenging computational science and engineering problems with little or no available physics as well as with extreme computational demand. The detailed construction and mathematical elements of HiDeNN are introduced and discussed to show the flexibility of the framework for diverse problems from disparate fields. Three example problems are solved to demonstrate the accuracy, efficiency, and versatility of the framework. The first example is designed to show that HiDeNN is capable of achieving better accuracy than conventional finite element method by learning the optimal nodal positions and capturing the stress concentration with a coarse mesh. The second example applies HiDeNN for multiscale analysis with sub-neural networks at each material point of macroscale. The final example demonstrates how HiDeNN can discover governing dimensionless parameters from experimental data so that a reduced set of input can be used to increase the learning efficiency. We further present a discussion and demonstration of the solution for advanced engineering problems that require state-of-the-art AI approaches and how a general and flexible system, such as HiDeNN-AI framework, can be applied to solve these problems.

Introduction

With the great development of modern computer and computational algorithms, computational science and engineering have achieved enormous success in almost all fields, such as physics, chemistry, biology, mechanical, civil, and materials science and engineering. However, many problems in computational science across the disciplines are still challenging. We propose that there are three major classes, or types, of problems puzzling the community of computational science and engineering. These three types are:

  • 1.

    Type 1 or purely data-driven problems: The class of analyses with unknown or still developing governing physics but abundant data. For these problems, the lack of knowledge of physics can be compensated by the presence of considerable data from carefully designed experiments regarding the system response.

  • 2.

    Type 2 or mechanistically insufficient problems with limited data: The term mechanistic refers to the theories which explain a phenomenon in purely physical or deterministic terms [1]. Type 2 problems are characterized by physical equations that require complementary data to provide a complete solution.

  • 3.

    Type 3 or computationally expensive problems: The problems for which the governing equations are known but too computationally burdensome to solve.

We will attempt to show that artificial intelligence (AI), particularly a subset of AI, deep learning, is a promising way to solve these challenging problems. An AI system is identified by its capability to perform tasks which currently humans perform in a better way [2]. This is famously judged by the Turing test, proposed to measure the intelligence of a machine by its capability to imitate human behavior [3]. An AI system can be classified into three classes, (a) “weak” or narrow AI, (b) general AI, and (c) super AI [4]. A narrow or “weak” AI is designed to perform a specific task and outperform any human in doing that. General AI refers to an AI system that may exhibit intelligent behavior in different areas and might outperform humans [5]. Super AI is a conceptual version of the technology that is the supreme point where machine achieves superhuman intelligence and can perform abstract thinking [6]. Almost all of the AI systems we see around us fall in the category of narrow AI. Super and general AI are still futuristic ideas. Machine learning (ML) is a form of narrow AI [7] and defined as the process by which computers, when given data, create their own knowledge (hence the term learning) by identifying patterns in data [8], [9]. Deep neural network is a subset of machine learning tools by which computers “understand” challenging and complex concepts by building the deep hierarchy of simpler concepts [9]. A generic deep neural network consists of input layer, hidden layers, and output layer where the input (layer) is connected (nonlinear information processing) through an activation function (hidden layer) to the output (layer) [8].

There is a growing tendency across the scientific communities to engage narrow AI (machine learning or deep learning) to solve problems in disciplines such as mechanics [10], [11], [12], biology and bio-medicine [13], [14], [15], materials science and engineering [16], [17], [18], manufacturing process monitoring [19], [20], [21], topology optimization [22], [23], [24], design under uncertainty [25], and miscellaneous engineering disciplines [26], [27], [28]. The scope of machine learning tools to aid or solve computational science problems goes beyond merely regressing non-linear data. Deep neural network and transfer learning are now being applied to discover hidden governing physical laws from data [29], [30], [31], speed up the computation in multiscale and multiphysics problems [32], [33], [34], [35], [36], characterize and reconstruct complex microstructures [37], design of heterogeneous materials and metamaterials [38], [39], discover new materials [40], [41], and to model path- and history-dependent problems [42], [43], [44], [45]. Fig. 1 shows AI tools currently in use to solve state-of-art computational science problems. The AI tools include data generation and collection techniques, feature extraction techniques (wavelet and Fourier transform [46], principal component analysis [46]), dimension reduction techniques (clustering, self-organizing map [21], [46]), regression (neural network, random forest) [46], reduced order models (can be something similar to regression techniques or more advanced technique like self-consistent clustering analysis (SCA) [47], [48] or Proper Orthogonal Decomposition (POD) [46]) and classification (convolutional neural networks or CNN [46]).

There are several practical challenges in directly applying current AI frameworks to solve aforementioned types of problems: (1) it is often difficult to decide on the criteria to identify the type of the problem and on the set of tools to use; (2) for a computational materials scientist or practicing engineer, it might become a challenging task to go back and forth among the different machine learning tools; (3) a design engineer needs to have a closed form relationship among different parameters controlling the desired property of the system. Moreover, the bridge connecting seemingly disparate fields of data-science and computational methods has to be a general one so that a common framework can be used to solve problems of different nature and originating from different physics. One other problem for applying AI frameworks in science and engineering is the paucity of data. Often experiments are too expensive to be useful to generate a large amount of data. Computational and theoretical predictions are limited by inherent assumptions. Considering these current difficulties and constraints, we propose a unified deep learning framework named Hierarchical Deep Learning Neural Network (HiDeNN). An advantage of using the HiDeNN structure is that such a neural network has a universal approximation capability enabling it to correctly interpolate among the data points generated by extremely non-linear relationships. HiDeNN can identify the governing physics from an experimental dataset without any prior knowledge and therefore can fill the missing link between data and mechanistic knowledge. As will be explained, HiDeNN also has the capability to incorporate mechanistic knowledge in training through proper definition of the loss function. This will reduce the necessity of a large amount of data to get an accurate prediction. A practical example is provided in a companion paper by Tajdari et al. [49], submitted to the same issue, to demonstrate in detail how a small amount of medical data available for adolescent idiopathic scoliosis can be used with mechanistic knowledge and deep learning to predict spine curvature. All the machine learning tools and computational methods mentioned earlier can be built into HiDeNN, eliminating the need for the user to decide on specific tools.

This article is organized as follows: Section 2 introduces and describes the components of HiDeNN, Section 3 presents the application of HiDeNN framework by solving three illustrative problems, Section 4 discusses three examples from each type of challenging problems, how those are solved using state-of-the-art methods, and recast the solution of the problems using HiDeNN, Section 5 proposes possible extensions of HiDeNN for general problems.

Section snippets

Hierarchical deep learning neural network (HiDeNN)

An example structure of HiDeNN for a general computational science and engineering problem is shown in Fig. 2. Construction of HiDeNN framework is discussed in following points:

  • The input layer of HiDeNN consists of inputs from spatial (Ω), temporal (t), and parameter (D) spaces. The neurons of this layer serve as independent variables of any physical system.

  • The input layer of HiDeNN is connected to a set of neurons that represents a set of pre-processing functions f(x,t,p) where x,t, and p are

Application of HiDeNN framework

In this section, three examples of HiDeNN are discussed in detail to demonstrate the framework’s capability.

Extension of HiDeNN to solve challenging problems

This section demonstrates a typical AI solution method for one example of each type of the challenging problems introduced in Section 1, and make note of challenges with these existing methods that might be mitigated by using HiDeNN.

Future outlooks of HiDeNN

The article so far discusses the construction and application of HiDeNN framework and how we can apply the framework to three challenging problems (see Section 4) in computational science and engineering. This section discusses on necessary future extensions of HiDeNN.

To solve type 2 or mechanistically insufficient problems with limited data, we might need to leverage the available experimental data from literature. However, the data coming from multiple sources are bound to suffer from lack of

Summary

We present a novel framework, HiDeNN, as a narrow AI methodology to solve a variety of computational science and engineering problems. HiDeNN can assimilate many data-driven tools in an appropriate way, which provides a general approach to solve challenging computational problems from different fields. A detailed discussion on the construction of HiDeNN highlights the flexibility and generality of this framework. We illustrate an application of HiDeNN to perform multiscale analysis of composite

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

The authors would like to acknowledge the support of National Science Foundation (NSF, USA) grants CMMI-1762035 and CMMI-1934367 and AFOSR, USA grant FA9550-18-1-0381. We thank Jennifer Bennett and her academic adviser Jian Cao for providing experimental data for Section 4.1.

References (81)

  • LiuY. et al.

    Materials discovery and design using machine learning

    J. Mater.

    (2017)
  • TangS. et al.

    Map123: A data-driven approach to use 1d data for 3d nonlinear elastic materials modeling

    Comput. Methods Appl. Mech. Engrg.

    (2019)
  • TangS. et al.

    Map123-ep: A mechanistic-based data-driven approach for numerical elastoplastic analysis

    Comput. Methods Appl. Mech. Engrg.

    (2020)
  • LiuZ. et al.

    Self-consistent clustering analysis: an efficient multi-scale scheme for inelastic heterogeneous materials

    Comput. Methods Appl. Mech. Engrg.

    (2016)
  • YuC. et al.

    Self-consistent clustering analysis for multiscale modeling at finite strains

    Comput. Methods Appl. Mech. Engrg.

    (2019)
  • HughesT. et al.

    Isogeometric analysis: Cad, finite elements, nurbs, exact geometry and mesh refinement

    Comput. Methods Appl. Mech. Engrg.

    (2005)
  • YangS. et al.

    Modeling of non-uniform spatial arrangement of fibers in a ceramic matrix composite

    Acta Mater.

    (1997)
  • FeyelF. et al.

    Fe2 multiscale approach for modelling the elastoviscoplastic behaviour of long fibre sic/ti composite materials

    Comput. Methods Appl. Mech. Engrg.

    (2000)
  • LiuZ. et al.

    Microstructural material database for self-consistent clustering analysis of elastoplastic strain softening materials

    Comput. Methods Appl. Mech. Engrg.

    (2018)
  • HornikK. et al.

    Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks

    Neural Netw.

    (1990)
  • BennettJ.L. et al.

    Cooling rate effect on tensile strength of laser deposited inconel 718

    Procedia Manuf.

    (2018)
  • GaoJ. et al.

    Predictive multiscale modeling for unidirectional carbon fiber reinforced polymers

    Compos. Sci. Technol.

    (2020)
  • StaffM.-W.

    Merriam-Webster’s Collegiate Dictionary, Vol. 2

    (2004)
  • RichE. et al.

    Artificial Intelligence

    (1990)
  • TuringA.M.

    Computing machinery and intelligence

  • SilverD. et al.

    Mastering the game of go without human knowledge

    Nature

    (2017)
  • GoertzelB. et al.

    A foundational architecture for artificial general intelligence

    Adv. Artif. Gen. Intell.: Concepts Archit. Algorithms

    (2007)
  • DengL. et al.

    Deep learning: methods and applications

    Found. Trends Signal Process.

    (2014)
  • GoodfellowI. et al.

    Deep Learning

    (2016)
  • IbañezR. et al.

    Data-driven non-linear elasticity: constitutive manifold construction and problem discretization

    Comput. Mech.

    (2017)
  • LiuW.K. et al.

    A Computational Mechanics Special Issue On: Data-Driven Modeling and Simulation—Theory, Methods, and Applications

    (2019)
  • TarcaA.L. et al.

    Machine learning and its applications to biology

    PLoS Comput. Biol.

    (2007)
  • HalabiS.S. et al.

    The rsna pediatric bone age machine learning challenge

    Radiology

    (2019)
  • WagnerN. et al.

    Theory-guided machine learning in materials science

    Front. Mater.

    (2016)
  • SchmidtJ. et al.

    Recent advances and applications of machine learning in solid-state materials science

    npj Comput. Mater.

    (2019)
  • UshizimaD. et al.

    Data science and machine learning for polymer films and beyond

    Bull. Amer. Phys. Soc.

    (2020)
  • YuY. et al.

    Deep learning for determining a near-optimal topological design without any iteration

    Struct. Multidiscip. Optim.

    (2019)
  • LeiX. et al.

    Machine learning-driven real-time topology optimization under moving morphable component-based framework

    J. Appl. Mech.

    (2019)
  • SosnovikI. et al.

    Neural networks for topology optimization

    Russ. J. Numer. Anal. Math. Modelling

    (2019)
  • TabarY.R. et al.

    A novel deep learning approach for classification of eeg motor imagery signals

    J. Neural Eng,

    (2016)
  • Cited by (100)

    View all citing articles on Scopus
    1

    Sourav Saha, Zhengtao Gan, and Lin Cheng contributed equally to this work..

    2

    Current address: Applied Chemicals and Materials Division, National Institute of Standards and Technology, Boulder, CO 80305, USA.

    View full text