Non-intrusive reduced-order modeling using uncertainty-aware Deep Neural Networks and Proper Orthogonal Decomposition: Application to flood modeling
Introduction
Machine Learning and other forms of Artificial Intelligence (AI) have been at the epicenter of massive breakthroughs in the notoriously difficult fields of computer vision, language modeling and content generation, as presented in [1], [2], and [3]. Still, there are many other domains where robust and well-tested methods could be significantly improved by the modern computational tools associated with AI: antibiotic discovery is just one very recent example [4]. In the realm of high-fidelity computational mechanics, simulation time is tightly linked to the size of the mesh and the number of time-steps; in other words, to its accuracy, which could make it impractical to be used in real-time contexts for new parameters.
Much research has been performed to address this large-size problem and to create Reduced-Ordered Models (ROM) that can effectively replace their heavier counterpart for tasks like design and optimization, or for real-time predictions. The most common way to build a ROM is to go through a compression phase into a reduced space, defined by a set of reduced basis (RB) vectors, which is at the root of many methods, according to [5]. For the most part, RB methods involve an offline-online paradigm, where the former is more computationally-heavy, and the latter should be fast enough to allow for real-time predictions. The idea is to collect data points called snapshots from simulations, or any high-fidelity source, and extract the information that has the most significance on the dynamics of the system, the modes, via a reduction method in the offline stage.
Proper Orthogonal Decomposition (POD), as introduced in [6], [7], coupled with the Singular Value Decomposition (SVD) algorithm [8], is by far the most popular method to reach a low-rank approximation. Subsequently, the online stage involves recovering the expansion coefficients, projecting back into the uncompressed, real-life space. This recovery is where the separation between intrusive and non-intrusive methods appears, where the former use techniques based on the problem's formulation, such as the Galerkin procedure [9], [10], [11]. At the same time, the latter (non-intrusive methods) try to statistically infer the mapping by considering the snapshots as a dataset. In this non-intrusive context, the POD-NN framework proposed by [12] and extended for time-dependent problems in [13] aims at training an artificial Neural Network to perform the mapping. These time-dependent problems can also benefit from approaching the POD on a temporal subdomain level, which has proved useful to prevent long-term error propagation, as first detailed in [14] and later assessed in [11].
Conventionally, laws of physics are expressed as well-defined Partial Differential Equations (PDEs), with boundary/initial conditions as constraints. Still, lately, pure data-driven methods have led to new approaches in PDE discovery [15]. The explosive growth of this new field of Deep Learning in Computational Fluid Dynamics was predicted in [16]. Its flexibility allows for multiple applications, such as the recovery of missing CFD data [17], or aerodynamic design optimization [18]. The cost associated with a fine mesh is high, but this has been overcome with a Machine Learning (ML) approach aimed at assessing errors and correcting quantities in a coarser setting [19]. New research in the field of numerical schemes was performed in [20], presenting the Volume of Fluid-Machine Learning (VOF-ML) approach applied in bi-material settings. A review of the vast landscape of possibilities is offered in [21]. The constraints of small data also led researchers to try to balance the need for data in AI contexts with expert knowledge, as with governing equations. First presented in [22], this was then extended to neural networks in [23] with applications in Computational Fluid Dynamics, as well as in vibration analysis [24]. When modeling data organized in sequence, Recurrent Neural Networks [25] are often predominant, especially the Long Short Term Memory (LSTM) variant [26]. LSTM neural networks have recently been applied in the context of time-dependent flooding prediction in [27], with the promise of providing real-time results. A recent contribution by [28] even allows for an embedded Bayesian treatment. Finally, an older but thorough study of available Machine Learning methods applied to environmental sciences and hydrology is presented in [29].
While their regression power is impressive, Deep Neural Networks are still, in their standard state, only able to predict a mean value, and do not provide any guidance on how much trust one can put into that value. To address this, recent additions to the Machine Learning landscape include Deep Ensembles [30] which suggest the training of an ensemble of specific, variance-informed deep neural networks, to obtain a complete uncertainty treatment. That work was subsequently extended to sub-ensembles for faster implementation in [31] and later reviewed in [32]. Earlier, other works had successfully encompassed the Bayesian view of probabilities within a Deep Neural Network, with the work of [33], [34], [35], [36] ultimately leading to the backpropagation-compatible Bayesian Neural Networks defined in [37], making use of Variational Inference [38], and paving the way for trainable Bayesian Neural Networks, also reviewed in [32]. Notable applications are surrogate modeling for inverse problems [39], and Bayesian physics-informed neural networks [40].
In this work, we aim at transferring the recent breakthroughs in Deep Learning to Computational Fluid Dynamics, by extending the concept of POD-NN with state-of-the-art methods for uncertainty quantification in Deep Neural Networks. After setting up the POD approach in Section 2, the methodologies of Deep Ensembles and Variational Inference for Bayesian Neural Networks are presented in Sections 3 and 4, respectively. Their performances are assessed according to a benchmark in Section 5. Our context of interest, flood modeling, is addressed in Section 6. A dam break scenario is presented in Section 6.2, first in a 1D Riemann analytically tractable example in order to obtain a reproducible problem in this context and to validate the numerical solver used in higher-dimension problems. The primary engineering aim is the training of a model capable of producing probabilistic flooding maps of the river presented in Section 6.3.1, with its results reported in Section 6.3.2. A contribution to standard uncertainty propagation is offered in 6.3.3, while Section 6.4 uses the same river environment for a fictitious dam break simulation. The Mille Îles river located in the Greater Montreal area is considered for these real-life application examples. We summarize our conclusions on this successful application of Deep Ensembles and Variational Inference for Bayesian Neural Networks in Section 7, along with our recommendations for the most promising future work in this area.
Section snippets
Objective and setup
We start by defining u, our -valued function of interest with as the spatial parameters and as the additional non-spatial parameters, for anything from a fluid viscosity to the time variable.
Computing this function is costly, so only a finite number S of solutions called snapshots can be realized. These are obtained over a discretized space, which can either be a uniform grid or an unstructured mesh, with n representing the number of dimensions and D the number
Regression objective
Building a non-intrusive ROM involves a statistical step to construct the function responsible for inferring the expansion parameters v from new non-spatial parameters s. This regression step is performed offline, and as we have considered the spatial parameters x to be externally handled, it can be represented as a mapping outputting the projection coefficients , as in
Deep Neural Networks with built-in variance
This statistical step is handled in the POD-NN framework by inferring the mapping with a Deep Neural
Bayesian Neural Networks and variational inference as an alternative
Making a model aware of its associated uncertainties can ultimately be achieved by adopting the Bayesian view. Recently, it has become easier to include a fully Bayesian treatment within Deep Neural Networks [37], designed to be compatible with backpropagation. In this section, we implement this version of Bayesian Neural Networks within the POD-NN framework, which we will refer to as POD-BNN, and compare it to the Deep Ensembles approach.
Benchmark with uncertainty quantification
In this section, we assess the uncertainty propagation component of our framework against a steady and two-dimensional benchmark, known as the Ackley Function.
Flood modeling application: the Mille Îles river
After assessing how both the Deep Ensembles and the BNN version of the POD-NN model performed on a 2D benchmark problem, here we aim at a real-world engineering problem: flood modeling. The goal is to propose a methodology to predict probabilistic flood maps. Quantification of the uncertainties in the flood zones is assessed through the propagation of the input parameters' aleatoric uncertainties via the numerical solver of the Shallow Water equations.
Conclusion
The excellent regression power of Deep Neural Networks has proved to be an asset to deploy along with Proper Orthogonal Decomposition to build reduced-order models. Their advantage is most notable when extended with recent progress in Deep Learning for a Computational Fluid Dynamics application.
Utilizing 1D and 2D benchmarks, we have shown that this approach achieved excellent results in terms of accuracy, and the training times were very reasonable, even on regular computers. It combines
CRediT authorship contribution statement
Pierre Jacquier: Conceptualization, Methodology, Software, validation, Writing - original draft preparation. Azedine Abdedou: Conceptualization, Methodology, Validation. Vincent Delmas: Investigation, Validation, Visualization. Azzeddine Soulaïmani: Conceptualization, Methodology, Reviewing, Supervision, Project administration, Funding.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
This research was enabled in part by funding from the Natural Sciences and Engineering Research Council of Canada and Hydro-Québec, by bathymetry data from the Montreal Metropolitan Community (Communauté métropolitaine de Montréal), and by computational support from Calcul Québec and Compute Canada.
References (65)
- et al.
A deep learning approach to antibiotic discovery
Cell
(2020) - et al.
Low-dimensional models of coherent structures in turbulence
Phys. Rep.
(1997) - et al.
Calibrated reduced-order POD-Galerkin system for fluid flow modelling
J. Comput. Phys.
(2005) - et al.
A POD-based reduced-order model for free surface shallow water flows over real bathymetries for Monte-Carlo-type applications
Comput. Methods Appl. Mech. Eng.
(2012) - et al.
Non-intrusive reduced order modeling of nonlinear problems using neural networks
J. Comput. Phys.
(2018) - et al.
Non-intrusive reduced order modeling of unsteady flows using artificial neural networks with application to a combustion problem
J. Comput. Phys.
(2019) - et al.
Recovering missing CFD data for high-order discretizations using deep neural networks and dynamics learning
J. Comput. Phys.
(2019) - et al.
Application of deep learning based multi-fidelity surrogate model to robust aerodynamic design optimization
Aerosp. Sci. Technol.
(2019) - et al.
Machine-learning based error prediction approach for coarse-grid Computational Fluid Dynamics (CG-CFD)
Prog. Nucl. Energy
(2020) - et al.
Machine Learning design of volume of Fluid schemes for compressible flows
J. Comput. Phys.
(2020)
Machine learning of linear differential equations using Gaussian processes
J. Comput. Phys.
Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
J. Comput. Phys.
Rapid spatio-temporal flood prediction and uncertainty quantification using a deep learning method
J. Hydrol.
TELEMAC: a new numerical model for solving shallow water equations
Adv. Water Resour.
Inception-v4, inception-ResNet and the impact of residual connections on learning
Distributed representations of words and phrases and their compositionality
Analyzing and improving the image quality of StyleGAN
A survey of projection-based model reduction methods for parametric dynamical systems
SIAM Rev.
Turbulence and the dynamics of coherent structures. I. Coherent structures
Q. Appl. Math.
Centroidal Voronoi tessellation-based reduced-order modeling of complex systems
SIAM J. Sci. Comput.
A POD-based reduced-order model for uncertainty analyses in shallow water flows
Int. J. Comput. Fluid Dyn.
Signal Representation and Modeling of Spatial Structures in Fluids
Discovering governing equations from data by sparse identification of nonlinear dynamical systems
Proc. Natl. Acad. Sci. USA
Deep learning in fluid dynamics
J. Fluid Mech.
Data-Driven Science and Engineering
Deep learning of vortex-induced vibrations
J. Fluid Mech.
Learning internal representations by error propagation
Long short-term memory
Neural Comput.
Bayesian recurrent neural network models for forecasting and quantifying uncertainty in spatial-temporal data
Entropy
Machine Learning Methods in the Environmental Sciences: Neural Networks and Kernels
Simple and scalable predictive uncertainty estimation using deep ensembles
Deep Sub-Ensembles for Fast Uncertainty Estimation in Image Classification
Cited by (42)
Reduced-order model-based variational inference with normalizing flows for Bayesian elliptic inverse problems
2024, Journal of Computational and Applied MathematicsMulti-fidelity nonlinear unsteady aerodynamic modeling and uncertainty estimation based on Hierarchical Kriging
2023, Applied Mathematical ModellingDeep learning enabled surrogate model of complex food processes for rapid prediction
2023, Chemical Engineering ScienceData coverage assessment on neural network based digital twins for autonomous control system
2023, Annals of Nuclear EnergyCitation Excerpt :The prediction error increases as the dissimilarity characterized by the minimum DTW distance becomes larger. When NNs are used for flooding modeling, the predictive uncertainty can detect data that the network does not know (Jacquier et al., 2021), and it is shown that the mean prediction interval width is larger for the unseen data. Uncertainty-based methods have been successfully used for similar tasks, including some engineering applications (Palacci and Hess, 2018; Tagasovska and Lopez-Paz, 2019; Ovadia et al., 2019; Combalia et al., 2020; Jacquier et al., 2021; Zhou et al., 2022).
Physics-informed deep learning: A promising technique for system reliability assessment
2022, Applied Soft ComputingCitation Excerpt :Most notably, physics-informed generative adversarial networks (PIGANs) have been proposed and are still under development to probabilistically leverage observations and the underlying physics model in solving both forward and inverse problems. The studies of PIGANs mainly use a similar framework and vary depending on two factors: integrate domain knowledge into either generator or discriminator [29]; adopt different types of GANs to improve their training stability [20,21,30–32]. The choice of PIGANs would vary depending on the scale of specific problems and the computational resources available.