Non-intrusive reduced-order modeling using uncertainty-aware Deep Neural Networks and Proper Orthogonal Decomposition: Application to flood modeling

https://doi.org/10.1016/j.jcp.2020.109854Get rights and content

Highlights

  • An uncertainty-aware Proper Orthogonal Decomposition and Neural Networks framework.

  • Comparison of Ensembles (POD-EnsNN) and Bayesian (POD-BNN) non-linear regression.

  • Ability to present a large-uncertainty warning on “out-of-distribution” predictions.

  • Application to probabilistic flooding maps generation, as well as to dam-break flows.

Abstract

Deep Learning research is advancing at a fantastic rate, and there is much to gain from transferring this knowledge to older fields like Computational Fluid Dynamics in practical engineering contexts. This work compares state-of-the-art methods that address uncertainty quantification in Deep Neural Networks, pushing forward the reduced-order modeling approach of Proper Orthogonal Decomposition-Neural Networks (POD-NN) with Deep Ensembles and Variational Inference-based Bayesian Neural Networks on two-dimensional problems in space. These are first tested on benchmark problems, and then applied to a real-life application: flooding predictions in the Mille Îles river in the Montreal, Quebec, Canada metropolitan area. Our setup involves a set of input parameters, with a potentially noisy distribution, and accumulates the simulation data resulting from these parameters. The goal is to build a non-intrusive surrogate model that is able to know when it does not know, which is still an open research area in Neural Networks (and in AI in general). With the help of this model, probabilistic flooding maps are generated, aware of the model uncertainty. These insights on the unknown are also utilized for an uncertainty propagation task, allowing for flooded area predictions that are broader and safer than those made with a regular uncertainty-uninformed surrogate model. Our study of the time-dependent and highly nonlinear case of a dam break is also presented. Both the ensembles and the Bayesian approach lead to reliable results for multiple smooth physical solutions, providing the correct warning when going out-of-distribution. However, the former, referred to as POD-EnsNN, proved much easier to implement and showed greater flexibility than the latter in the case of discontinuities, where standard algorithms may oscillate or fail to converge.

Introduction

Machine Learning and other forms of Artificial Intelligence (AI) have been at the epicenter of massive breakthroughs in the notoriously difficult fields of computer vision, language modeling and content generation, as presented in [1], [2], and [3]. Still, there are many other domains where robust and well-tested methods could be significantly improved by the modern computational tools associated with AI: antibiotic discovery is just one very recent example [4]. In the realm of high-fidelity computational mechanics, simulation time is tightly linked to the size of the mesh and the number of time-steps; in other words, to its accuracy, which could make it impractical to be used in real-time contexts for new parameters.

Much research has been performed to address this large-size problem and to create Reduced-Ordered Models (ROM) that can effectively replace their heavier counterpart for tasks like design and optimization, or for real-time predictions. The most common way to build a ROM is to go through a compression phase into a reduced space, defined by a set of reduced basis (RB) vectors, which is at the root of many methods, according to [5]. For the most part, RB methods involve an offline-online paradigm, where the former is more computationally-heavy, and the latter should be fast enough to allow for real-time predictions. The idea is to collect data points called snapshots from simulations, or any high-fidelity source, and extract the information that has the most significance on the dynamics of the system, the modes, via a reduction method in the offline stage.

Proper Orthogonal Decomposition (POD), as introduced in [6], [7], coupled with the Singular Value Decomposition (SVD) algorithm [8], is by far the most popular method to reach a low-rank approximation. Subsequently, the online stage involves recovering the expansion coefficients, projecting back into the uncompressed, real-life space. This recovery is where the separation between intrusive and non-intrusive methods appears, where the former use techniques based on the problem's formulation, such as the Galerkin procedure [9], [10], [11]. At the same time, the latter (non-intrusive methods) try to statistically infer the mapping by considering the snapshots as a dataset. In this non-intrusive context, the POD-NN framework proposed by [12] and extended for time-dependent problems in [13] aims at training an artificial Neural Network to perform the mapping. These time-dependent problems can also benefit from approaching the POD on a temporal subdomain level, which has proved useful to prevent long-term error propagation, as first detailed in [14] and later assessed in [11].

Conventionally, laws of physics are expressed as well-defined Partial Differential Equations (PDEs), with boundary/initial conditions as constraints. Still, lately, pure data-driven methods have led to new approaches in PDE discovery [15]. The explosive growth of this new field of Deep Learning in Computational Fluid Dynamics was predicted in [16]. Its flexibility allows for multiple applications, such as the recovery of missing CFD data [17], or aerodynamic design optimization [18]. The cost associated with a fine mesh is high, but this has been overcome with a Machine Learning (ML) approach aimed at assessing errors and correcting quantities in a coarser setting [19]. New research in the field of numerical schemes was performed in [20], presenting the Volume of Fluid-Machine Learning (VOF-ML) approach applied in bi-material settings. A review of the vast landscape of possibilities is offered in [21]. The constraints of small data also led researchers to try to balance the need for data in AI contexts with expert knowledge, as with governing equations. First presented in [22], this was then extended to neural networks in [23] with applications in Computational Fluid Dynamics, as well as in vibration analysis [24]. When modeling data organized in sequence, Recurrent Neural Networks [25] are often predominant, especially the Long Short Term Memory (LSTM) variant [26]. LSTM neural networks have recently been applied in the context of time-dependent flooding prediction in [27], with the promise of providing real-time results. A recent contribution by [28] even allows for an embedded Bayesian treatment. Finally, an older but thorough study of available Machine Learning methods applied to environmental sciences and hydrology is presented in [29].

While their regression power is impressive, Deep Neural Networks are still, in their standard state, only able to predict a mean value, and do not provide any guidance on how much trust one can put into that value. To address this, recent additions to the Machine Learning landscape include Deep Ensembles [30] which suggest the training of an ensemble of specific, variance-informed deep neural networks, to obtain a complete uncertainty treatment. That work was subsequently extended to sub-ensembles for faster implementation in [31] and later reviewed in [32]. Earlier, other works had successfully encompassed the Bayesian view of probabilities within a Deep Neural Network, with the work of [33], [34], [35], [36] ultimately leading to the backpropagation-compatible Bayesian Neural Networks defined in [37], making use of Variational Inference [38], and paving the way for trainable Bayesian Neural Networks, also reviewed in [32]. Notable applications are surrogate modeling for inverse problems [39], and Bayesian physics-informed neural networks [40].

In this work, we aim at transferring the recent breakthroughs in Deep Learning to Computational Fluid Dynamics, by extending the concept of POD-NN with state-of-the-art methods for uncertainty quantification in Deep Neural Networks. After setting up the POD approach in Section 2, the methodologies of Deep Ensembles and Variational Inference for Bayesian Neural Networks are presented in Sections 3 and 4, respectively. Their performances are assessed according to a benchmark in Section 5. Our context of interest, flood modeling, is addressed in Section 6. A dam break scenario is presented in Section 6.2, first in a 1D Riemann analytically tractable example in order to obtain a reproducible problem in this context and to validate the numerical solver used in higher-dimension problems. The primary engineering aim is the training of a model capable of producing probabilistic flooding maps of the river presented in Section 6.3.1, with its results reported in Section 6.3.2. A contribution to standard uncertainty propagation is offered in 6.3.3, while Section 6.4 uses the same river environment for a fictitious dam break simulation. The Mille Îles river located in the Greater Montreal area is considered for these real-life application examples. We summarize our conclusions on this successful application of Deep Ensembles and Variational Inference for Bayesian Neural Networks in Section 7, along with our recommendations for the most promising future work in this area.

Section snippets

Objective and setup

We start by defining u, our RD-valued function of interestu:Rn+PRD(x,s)u(x,s), with xRn as the spatial parameters and sRP as the additional non-spatial parameters, for anything from a fluid viscosity to the time variable.

Computing this function is costly, so only a finite number S of solutions called snapshots can be realized. These are obtained over a discretized space, which can either be a uniform grid or an unstructured mesh, with n representing the number of dimensions and D the number

Regression objective

Building a non-intrusive ROM involves a statistical step to construct the function responsible for inferring the expansion parameters v from new non-spatial parameters s. This regression step is performed offline, and as we have considered the spatial parameters x to be externally handled, it can be represented as a mapping uDB outputting the projection coefficients v(s), as inuDB:RPRLsv(s).

Deep Neural Networks with built-in variance

This statistical step is handled in the POD-NN framework by inferring the mapping with a Deep Neural

Bayesian Neural Networks and variational inference as an alternative

Making a model aware of its associated uncertainties can ultimately be achieved by adopting the Bayesian view. Recently, it has become easier to include a fully Bayesian treatment within Deep Neural Networks [37], designed to be compatible with backpropagation. In this section, we implement this version of Bayesian Neural Networks within the POD-NN framework, which we will refer to as POD-BNN, and compare it to the Deep Ensembles approach.

Benchmark with uncertainty quantification

In this section, we assess the uncertainty propagation component of our framework against a steady and two-dimensional benchmark, known as the Ackley Function.

Flood modeling application: the Mille Îles river

After assessing how both the Deep Ensembles and the BNN version of the POD-NN model performed on a 2D benchmark problem, here we aim at a real-world engineering problem: flood modeling. The goal is to propose a methodology to predict probabilistic flood maps. Quantification of the uncertainties in the flood zones is assessed through the propagation of the input parameters' aleatoric uncertainties via the numerical solver of the Shallow Water equations.

Conclusion

The excellent regression power of Deep Neural Networks has proved to be an asset to deploy along with Proper Orthogonal Decomposition to build reduced-order models. Their advantage is most notable when extended with recent progress in Deep Learning for a Computational Fluid Dynamics application.

Utilizing 1D and 2D benchmarks, we have shown that this approach achieved excellent results in terms of accuracy, and the training times were very reasonable, even on regular computers. It combines

CRediT authorship contribution statement

Pierre Jacquier: Conceptualization, Methodology, Software, validation, Writing - original draft preparation. Azedine Abdedou: Conceptualization, Methodology, Validation. Vincent Delmas: Investigation, Validation, Visualization. Azzeddine Soulaïmani: Conceptualization, Methodology, Reviewing, Supervision, Project administration, Funding.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This research was enabled in part by funding from the Natural Sciences and Engineering Research Council of Canada and Hydro-Québec, by bathymetry data from the Montreal Metropolitan Community (Communauté métropolitaine de Montréal), and by computational support from Calcul Québec and Compute Canada.

References (65)

  • M. Raissi et al.

    Machine learning of linear differential equations using Gaussian processes

    J. Comput. Phys.

    (2017)
  • M. Raissi et al.

    Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations

    J. Comput. Phys.

    (2019)
  • R. Hu et al.

    Rapid spatio-temporal flood prediction and uncertainty quantification using a deep learning method

    J. Hydrol.

    (2019)
  • J.-C. Galland et al.

    TELEMAC: a new numerical model for solving shallow water equations

    Adv. Water Resour.

    (1991)
  • C. Szegedy et al.

    Inception-v4, inception-ResNet and the impact of residual connections on learning

  • T. Mikolov et al.

    Distributed representations of words and phrases and their compositionality

  • T. Karras et al.

    Analyzing and improving the image quality of StyleGAN

  • P. Benner et al.

    A survey of projection-based model reduction methods for parametric dynamical systems

    SIAM Rev.

    (2015)
  • L. Sirovich

    Turbulence and the dynamics of coherent structures. I. Coherent structures

    Q. Appl. Math.

    (1987)
  • J. Burkardt et al.

    Centroidal Voronoi tessellation-based reduced-order modeling of complex systems

    SIAM J. Sci. Comput.

    (2006)
  • J.M. Zokagoa et al.

    A POD-based reduced-order model for uncertainty analyses in shallow water flows

    Int. J. Comput. Fluid Dyn.

    (2018)
  • W. Ijzerman

    Signal Representation and Modeling of Spatial Structures in Fluids

    (2000)
  • S.L. Brunton et al.

    Discovering governing equations from data by sparse identification of nonlinear dynamical systems

    Proc. Natl. Acad. Sci. USA

    (2016)
  • J.N. Kutz

    Deep learning in fluid dynamics

    J. Fluid Mech.

    (2017)
  • S.L. Brunton et al.

    Data-Driven Science and Engineering

    (2019)
  • M. Raissi et al.

    Deep learning of vortex-induced vibrations

    J. Fluid Mech.

    (2019)
  • D.E. Rumelhart et al.

    Learning internal representations by error propagation

    (1985)
  • S. Hochreiter et al.

    Long short-term memory

    Neural Comput.

    (1997)
  • P.L. McDermott et al.

    Bayesian recurrent neural network models for forecasting and quantifying uncertainty in spatial-temporal data

    Entropy

    (2019)
  • W.W. Hsieh

    Machine Learning Methods in the Environmental Sciences: Neural Networks and Kernels

    (2009)
  • B. Lakshminarayanan et al.

    Simple and scalable predictive uncertainty estimation using deep ensembles

  • M. Valdenegro-Toro

    Deep Sub-Ensembles for Fast Uncertainty Estimation in Image Classification

    (2019)
  • Cited by (42)

    • Data coverage assessment on neural network based digital twins for autonomous control system

      2023, Annals of Nuclear Energy
      Citation Excerpt :

      The prediction error increases as the dissimilarity characterized by the minimum DTW distance becomes larger. When NNs are used for flooding modeling, the predictive uncertainty can detect data that the network does not know (Jacquier et al., 2021), and it is shown that the mean prediction interval width is larger for the unseen data. Uncertainty-based methods have been successfully used for similar tasks, including some engineering applications (Palacci and Hess, 2018; Tagasovska and Lopez-Paz, 2019; Ovadia et al., 2019; Combalia et al., 2020; Jacquier et al., 2021; Zhou et al., 2022).

    • Physics-informed deep learning: A promising technique for system reliability assessment

      2022, Applied Soft Computing
      Citation Excerpt :

      Most notably, physics-informed generative adversarial networks (PIGANs) have been proposed and are still under development to probabilistically leverage observations and the underlying physics model in solving both forward and inverse problems. The studies of PIGANs mainly use a similar framework and vary depending on two factors: integrate domain knowledge into either generator or discriminator [29]; adopt different types of GANs to improve their training stability [20,21,30–32]. The choice of PIGANs would vary depending on the scale of specific problems and the computational resources available.

    View all citing articles on Scopus
    View full text