Hierarchical Deep Learning Neural Network (HiDeNN): An artificial intelligence (AI) framework for computational science and engineering
Introduction
With the great development of modern computer and computational algorithms, computational science and engineering have achieved enormous success in almost all fields, such as physics, chemistry, biology, mechanical, civil, and materials science and engineering. However, many problems in computational science across the disciplines are still challenging. We propose that there are three major classes, or types, of problems puzzling the community of computational science and engineering. These three types are:
- 1.
Type 1 or purely data-driven problems: The class of analyses with unknown or still developing governing physics but abundant data. For these problems, the lack of knowledge of physics can be compensated by the presence of considerable data from carefully designed experiments regarding the system response.
- 2.
Type 2 or mechanistically insufficient problems with limited data: The term mechanistic refers to the theories which explain a phenomenon in purely physical or deterministic terms [1]. Type 2 problems are characterized by physical equations that require complementary data to provide a complete solution.
- 3.
Type 3 or computationally expensive problems: The problems for which the governing equations are known but too computationally burdensome to solve.
We will attempt to show that artificial intelligence (AI), particularly a subset of AI, deep learning, is a promising way to solve these challenging problems. An AI system is identified by its capability to perform tasks which currently humans perform in a better way [2]. This is famously judged by the Turing test, proposed to measure the intelligence of a machine by its capability to imitate human behavior [3]. An AI system can be classified into three classes, (a) “weak” or narrow AI, (b) general AI, and (c) super AI [4]. A narrow or “weak” AI is designed to perform a specific task and outperform any human in doing that. General AI refers to an AI system that may exhibit intelligent behavior in different areas and might outperform humans [5]. Super AI is a conceptual version of the technology that is the supreme point where machine achieves superhuman intelligence and can perform abstract thinking [6]. Almost all of the AI systems we see around us fall in the category of narrow AI. Super and general AI are still futuristic ideas. Machine learning (ML) is a form of narrow AI [7] and defined as the process by which computers, when given data, create their own knowledge (hence the term learning) by identifying patterns in data [8], [9]. Deep neural network is a subset of machine learning tools by which computers “understand” challenging and complex concepts by building the deep hierarchy of simpler concepts [9]. A generic deep neural network consists of input layer, hidden layers, and output layer where the input (layer) is connected (nonlinear information processing) through an activation function (hidden layer) to the output (layer) [8].
There is a growing tendency across the scientific communities to engage narrow AI (machine learning or deep learning) to solve problems in disciplines such as mechanics [10], [11], [12], biology and bio-medicine [13], [14], [15], materials science and engineering [16], [17], [18], manufacturing process monitoring [19], [20], [21], topology optimization [22], [23], [24], design under uncertainty [25], and miscellaneous engineering disciplines [26], [27], [28]. The scope of machine learning tools to aid or solve computational science problems goes beyond merely regressing non-linear data. Deep neural network and transfer learning are now being applied to discover hidden governing physical laws from data [29], [30], [31], speed up the computation in multiscale and multiphysics problems [32], [33], [34], [35], [36], characterize and reconstruct complex microstructures [37], design of heterogeneous materials and metamaterials [38], [39], discover new materials [40], [41], and to model path- and history-dependent problems [42], [43], [44], [45]. Fig. 1 shows AI tools currently in use to solve state-of-art computational science problems. The AI tools include data generation and collection techniques, feature extraction techniques (wavelet and Fourier transform [46], principal component analysis [46]), dimension reduction techniques (clustering, self-organizing map [21], [46]), regression (neural network, random forest) [46], reduced order models (can be something similar to regression techniques or more advanced technique like self-consistent clustering analysis (SCA) [47], [48] or Proper Orthogonal Decomposition (POD) [46]) and classification (convolutional neural networks or CNN [46]).
There are several practical challenges in directly applying current AI frameworks to solve aforementioned types of problems: (1) it is often difficult to decide on the criteria to identify the type of the problem and on the set of tools to use; (2) for a computational materials scientist or practicing engineer, it might become a challenging task to go back and forth among the different machine learning tools; (3) a design engineer needs to have a closed form relationship among different parameters controlling the desired property of the system. Moreover, the bridge connecting seemingly disparate fields of data-science and computational methods has to be a general one so that a common framework can be used to solve problems of different nature and originating from different physics. One other problem for applying AI frameworks in science and engineering is the paucity of data. Often experiments are too expensive to be useful to generate a large amount of data. Computational and theoretical predictions are limited by inherent assumptions. Considering these current difficulties and constraints, we propose a unified deep learning framework named Hierarchical Deep Learning Neural Network (HiDeNN). An advantage of using the HiDeNN structure is that such a neural network has a universal approximation capability enabling it to correctly interpolate among the data points generated by extremely non-linear relationships. HiDeNN can identify the governing physics from an experimental dataset without any prior knowledge and therefore can fill the missing link between data and mechanistic knowledge. As will be explained, HiDeNN also has the capability to incorporate mechanistic knowledge in training through proper definition of the loss function. This will reduce the necessity of a large amount of data to get an accurate prediction. A practical example is provided in a companion paper by Tajdari et al. [49], submitted to the same issue, to demonstrate in detail how a small amount of medical data available for adolescent idiopathic scoliosis can be used with mechanistic knowledge and deep learning to predict spine curvature. All the machine learning tools and computational methods mentioned earlier can be built into HiDeNN, eliminating the need for the user to decide on specific tools.
This article is organized as follows: Section 2 introduces and describes the components of HiDeNN, Section 3 presents the application of HiDeNN framework by solving three illustrative problems, Section 4 discusses three examples from each type of challenging problems, how those are solved using state-of-the-art methods, and recast the solution of the problems using HiDeNN, Section 5 proposes possible extensions of HiDeNN for general problems.
Section snippets
Hierarchical deep learning neural network (HiDeNN)
An example structure of HiDeNN for a general computational science and engineering problem is shown in Fig. 2. Construction of HiDeNN framework is discussed in following points:
- •
The input layer of HiDeNN consists of inputs from spatial (), temporal (), and parameter () spaces. The neurons of this layer serve as independent variables of any physical system.
- •
The input layer of HiDeNN is connected to a set of neurons that represents a set of pre-processing functions where , and are
Application of HiDeNN framework
In this section, three examples of HiDeNN are discussed in detail to demonstrate the framework’s capability.
Extension of HiDeNN to solve challenging problems
This section demonstrates a typical AI solution method for one example of each type of the challenging problems introduced in Section 1, and make note of challenges with these existing methods that might be mitigated by using HiDeNN.
Future outlooks of HiDeNN
The article so far discusses the construction and application of HiDeNN framework and how we can apply the framework to three challenging problems (see Section 4) in computational science and engineering. This section discusses on necessary future extensions of HiDeNN.
To solve type 2 or mechanistically insufficient problems with limited data, we might need to leverage the available experimental data from literature. However, the data coming from multiple sources are bound to suffer from lack of
Summary
We present a novel framework, HiDeNN, as a narrow AI methodology to solve a variety of computational science and engineering problems. HiDeNN can assimilate many data-driven tools in an appropriate way, which provides a general approach to solve challenging computational problems from different fields. A detailed discussion on the construction of HiDeNN highlights the flexibility and generality of this framework. We illustrate an application of HiDeNN to perform multiscale analysis of composite
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
The authors would like to acknowledge the support of National Science Foundation (NSF, USA) grants CMMI-1762035 and CMMI-1934367 and AFOSR, USA grant FA9550-18-1-0381. We thank Jennifer Bennett and her academic adviser Jian Cao for providing experimental data for Section 4.1.
References (81)
- et al.
Siri, siri, in my hand: Who’s the fairest in the land? On the interpretations, illustrations, and implications of artificial intelligence
Bus. Horiz.
(2019) - et al.
Thirty years of artificial intelligence in medicine (aime) conferences: a review of research themes
Artif. Intell. Med.
(2015) - et al.
Data-driven computational mechanics
Comput. Methods Appl. Mech. Engrg.
(2016) - et al.
Prediction of spinal curve progression in adolescent idiopathic scoliosis using random forest regression
Comput. Biol. Med.
(2018) - et al.
Tool life predictions in milling using spindle power with the neural network technique
J. Manuf. Process.
(2016) - et al.
In-process monitoring of porosity during laser additive manufacturing process
Addit. Manuf.
(2019) - et al.
Data-driven microstructure and microhardness design in additive manufacturing using a self-organizing map
Engineering
(2019) - et al.
A framework for data-driven analysis of materials under uncertainty: countering the curse of dimensionality
Comput. Methods Appl. Mech. Engrg.
(2017) - et al.
Deep learning-based feature engineering methods for improved building energy prediction
Appl. Energy
(2019) - et al.
An inverse modeling approach for predicting filled rubber performance
Comput. Methods Appl. Mech. Engrg.
(2019)
Materials discovery and design using machine learning
J. Mater.
Map123: A data-driven approach to use 1d data for 3d nonlinear elastic materials modeling
Comput. Methods Appl. Mech. Engrg.
Map123-ep: A mechanistic-based data-driven approach for numerical elastoplastic analysis
Comput. Methods Appl. Mech. Engrg.
Self-consistent clustering analysis: an efficient multi-scale scheme for inelastic heterogeneous materials
Comput. Methods Appl. Mech. Engrg.
Self-consistent clustering analysis for multiscale modeling at finite strains
Comput. Methods Appl. Mech. Engrg.
Isogeometric analysis: Cad, finite elements, nurbs, exact geometry and mesh refinement
Comput. Methods Appl. Mech. Engrg.
Modeling of non-uniform spatial arrangement of fibers in a ceramic matrix composite
Acta Mater.
Fe2 multiscale approach for modelling the elastoviscoplastic behaviour of long fibre sic/ti composite materials
Comput. Methods Appl. Mech. Engrg.
Microstructural material database for self-consistent clustering analysis of elastoplastic strain softening materials
Comput. Methods Appl. Mech. Engrg.
Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks
Neural Netw.
Cooling rate effect on tensile strength of laser deposited inconel 718
Procedia Manuf.
Predictive multiscale modeling for unidirectional carbon fiber reinforced polymers
Compos. Sci. Technol.
Merriam-Webster’s Collegiate Dictionary, Vol. 2
Artificial Intelligence
Computing machinery and intelligence
Mastering the game of go without human knowledge
Nature
A foundational architecture for artificial general intelligence
Adv. Artif. Gen. Intell.: Concepts Archit. Algorithms
Deep learning: methods and applications
Found. Trends Signal Process.
Deep Learning
Data-driven non-linear elasticity: constitutive manifold construction and problem discretization
Comput. Mech.
A Computational Mechanics Special Issue On: Data-Driven Modeling and Simulation—Theory, Methods, and Applications
Machine learning and its applications to biology
PLoS Comput. Biol.
The rsna pediatric bone age machine learning challenge
Radiology
Theory-guided machine learning in materials science
Front. Mater.
Recent advances and applications of machine learning in solid-state materials science
npj Comput. Mater.
Data science and machine learning for polymer films and beyond
Bull. Amer. Phys. Soc.
Deep learning for determining a near-optimal topological design without any iteration
Struct. Multidiscip. Optim.
Machine learning-driven real-time topology optimization under moving morphable component-based framework
J. Appl. Mech.
Neural networks for topology optimization
Russ. J. Numer. Anal. Math. Modelling
A novel deep learning approach for classification of eeg motor imagery signals
J. Neural Eng,
Cited by (100)
Extended tensor decomposition model reduction methods: Training, prediction, and design under uncertainty
2024, Computer Methods in Applied Mechanics and EngineeringIsogeometric Convolution Hierarchical Deep-learning Neural Network: Isogeometric analysis with versatile adaptivity
2023, Computer Methods in Applied Mechanics and EngineeringReview of advanced road materials, structures, equipment, and detection technologies
2023, Journal of Road Engineering
- 1
Sourav Saha, Zhengtao Gan, and Lin Cheng contributed equally to this work..
- 2
Current address: Applied Chemicals and Materials Division, National Institute of Standards and Technology, Boulder, CO 80305, USA.