B-PINNs: Bayesian physics-informed neural networks for forward and inverse PDE problems with noisy data

doi:10.1016/j.jcp.2020.109913

Journal of Computational Physics

Volume 425, 15 January 2021, 109913

https://doi.org/10.1016/j.jcp.2020.109913 Get rights and content

Abstract

We propose a Bayesian physics-informed neural network (B-PINN) to solve both forward and inverse nonlinear problems described by partial differential equations (PDEs) and noisy data. In this Bayesian framework, the Bayesian neural network (BNN) combined with a PINN for PDEs serves as the prior while the Hamiltonian Monte Carlo (HMC) or the variational inference (VI) could serve as an estimator of the posterior. B-PINNs make use of both physical laws and scattered noisy measurements to provide predictions and quantify the aleatoric uncertainty arising from the noisy data in the Bayesian framework. Compared with PINNs, in addition to uncertainty quantification, B-PINNs obtain more accurate predictions in scenarios with large noise due to their capability of avoiding overfitting. We conduct a systematic comparison between the two different approaches for the B-PINNs posterior estimation (i.e., HMC or VI), along with dropout used for quantifying uncertainty in deep neural networks. Our experiments show that HMC is more suitable than VI with mean field Gaussian approximation for the B-PINNs posterior estimation, while dropout employed in PINNs can hardly provide accurate predictions with reasonable uncertainty. Finally, we replace the BNN in the prior with a truncated Karhunen-Loève (KL) expansion combined with HMC or a deep normalizing flow (DNF) model as posterior estimators. The KL is as accurate as BNN and much faster but this framework cannot be easily extended to high-dimensional problems unlike the BNN based framework.

Introduction

The state-of-the-art in data-driven modeling has advanced significantly recently in applications across different fields [1], [2], [3], [4], [5], due to the rapid development of machine learning and the explosive growth of available data collected from different sensors (e.g., satellites, cameras, etc.). In general, purely data-driven methods require a large amount of data in order to get accurate results [6]. As a powerful alternative, recently the data-driven solvers for partial differential equations (PDEs) have drawn an increasing attention due to their capability to encode the underlying physical laws in the form of PDEs and give relatively accurate predictions for the unknown terms with limited data. In the first case we need “big data” while in the second case we can learn from “small data” as we explicitly utilize the physical laws or more broadly a parametrization of the physics.

Two typical approaches are the Gaussian processes regression (GPR) for PDEs [7], and the physics-informed neural networks (PINNs) [6], [8]. Built upon the Bayesian framework with built-in mechanism for uncertainty quantification, GPR is one of the most popular data-driven methods. However, vanilla GPR has difficulties in handling the nonlinearities when applied to solve PDEs, leading to restricted applications. On the other hand, PINNs have shown effectiveness in both forward and inverse problems for a wide range of PDEs [9], [10], [11], [12], [13]. However, PINNs are not equipped with built-in uncertainty quantification, which may restrict their applications, especially for scenarios where the data are noisy.

In previous work, the physics-informed generative adversarial networks were employed to quantify parametric uncertainty [10], and also polynomial chaos expansions in conjunction with dropout were utilized to quantify total uncertainty [9]. In addition, approaches using the Bayesian inference for quantifying uncertainties of PDE problems have also been developed recently [14], [15], [16]. Note that (1) the methods developed in [14], [15], [16] focus on solving inverse PDE problems, and (2) the boundary conditions as well as the source terms are assumed to be known in [14], [15], [16]. However, we may only have access to sparse and noisy measurements on the boundary conditions and the source terms in real-world applications. In the present work, we propose a Bayesian physics-informed neural networks (B-PINNs) to solve linear or nonlinear PDEs with noisy data for both forward and inverse problems, see Fig. 1. We note that all the information of boundary conditions/sources terms comes from noisy measurements in the present approach. The uncertainties arising from the scattered noisy data could be naturally quantified due to the Bayesian framework [17]. B-PINNs consist of two parts: a parameterized surrogate model, i.e., a Bayesian neural network (BNN) with prior for the unknown terms in a PDE, and an approach for estimating the posterior distributions of the parameters in the surrogate model. In particular, we employ the Hamiltonian Monte Carlo (HMC) [18], [19] or the variational inference (VI) [20], [21] for estimation of the posterior distributions. In addition, we note that a non-Bayesian framework model, i.e., the dropout, has been used to quantify the uncertainty in deep neural networks, including the PINNs for solving PDEs [9], [22]. We will validate the proposed B-PINNs method and conduct a systematic comparison with the dropout for both the forward and inverse PDE problems given noisy data.

In addition to BNNs, the Karhunen-Loève expansion is also a widely used representation of a stochastic process. As an illustration, we further test the case using the truncated Karhunen-Loève as the surrogate model while we use HMC or the deep normalizing flow (DNF) models [23] for estimating the posterior in the Bayesian framework.

The rest of the paper is organized as follows: In Sec. 2, we present the B-PINNs algorithm for solving forward/inverse PDE problems with noisy data, including the BNNs for PDEs and posterior estimation methods, i.e., the HMC and VI, used in this paper. In Sec. 3, we compare the performance of the B-PINNs and dropout on the tasks of function approximation, forward PDE problems, and inverse PDE problems. In addition, we present comparisons between B-PINNs and PINNs as well as the KL for nonlinear forward/inverse PDEs in Secs. 4-5. We present a summary in Sec. 6. Furthermore, in Appendix A we present a study on the priors of BNNs, in Appendix B we give more details on the DNF models, in Appendix C we present an example of inverse problem where the unknown term is a function, and in Appendix D we give another example showing the effectiveness of the present method for extrapolation with the help of PDE as constraint.

Section snippets

B-PINNs: Bayesian physics-informed neural networks

We consider a general partial differential equation (PDE) of the form $N_{x} (u; λ) = f, x \in D, B_{x} (u; λ) = b, x \in Γ,$ where $N_{x}$ is a general differential operator, D is the d-dimensional physical domain, $u = u (x)$ is the solution of the PDE, and λ is the vector of parameters in the PDE. Also, $f = f (x)$ is the forcing term, and $B_{x}$ is the boundary condition operator acting on the domain boundary Γ. In forward problems λ is prescribed, and hence our goal is to infer the distribution of u at any x. In inverse problems, λ is

Results and discussion

In this section we present a systematic comparison among the B-PINNs with different posterior sampling methods, i.e., HMC (B-PINN-HMC) and VI (B-PINN-VI), as well as the dropout [9], [22] for 1D function approximation, and 1D/2D forward/inverse PDE problems.

In all the cases, we employ a neural network with 2 hidden layers, each with width of 50, for B-PINNs. The prior for θ is set as independent standard Gaussian distribution for each component. Such size of the neural network and the prior

Comparison with PINNs

In this section, we will conduct a comparison between the B-PINN-HMC and PINNs for the 1D inverse problem in Sec. 3.3.1. We employ the Adam optimizer with $l = 10^{- 3}, β_{1} = 0.9, β_{2} = 0.999$ to train the PINN, with the number of the training steps set as $200, 000$ . The results of the PINNs are shown in Fig. 12. Note that the PINNs cannot quantify uncertainties of the predictive results.

As shown in Fig. 12, the predicted u and f could fit all the training points. In the cases where the noise scale is as small

Comparison with the truncated Karhunen-Loève expansion

So far we have shown the effectiveness of B-PINNs in solving PDE problems. As we know, a neural network is extremely overparametrized. Hence, we want to investigate if we can use other models with less parameters for our surrogate model in the Bayesian framework. For example, we consider the Karhunen-Loève expansion, a widely used representation for a stochastic process in the following study.

Summary

There are many sources of uncertainty in data-driven PDE solvers, including aleatoric uncertainty associated with noisy data, epistemic uncertainty associated with unknown parameters, and model uncertainty associated with the type of PDE that models the target phenomena. In this paper, we address aleatoric uncertainty for solving forward and inverse PDE problems, based on noisy data associated with the solution, source terms and boundary conditions. In particular, we employ physics-informed

CRediT authorship contribution statement

Liu Yang & Xuhui Meng: Conceptualization, Methodology, Investigation, Coding, Writing - original draft, Writing - review & editing, Visualization. George Em Karniadakis: Conceptualization, Methodology, Investigation, Writing - original draft, Writing - review & editing, Supervision, Project administration, Funding acquisition.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgement

This work was supported by the PhILMS grant DE-SC0019453, OSD/AFOSR MURI grant FA9550-20-1-0358, OSD/ARO MURI grant W911NF-15-1-0562, and the NIH grant U01 HL142518.

References (38)

J. Berg et al.
Data-driven discovery of PDEs in complex datasets
J. Comput. Phys.
(2019)
M. Raissi et al.
Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
J. Comput. Phys.
(2019)
M. Raissi et al.
Machine learning of linear differential equations using Gaussian processes
J. Comput. Phys.
(2017)
D. Zhang et al.
Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems
J. Comput. Phys.
(2019)
X. Meng et al.
A composite neural network that learns from multi-fidelity data: application to function approximation and inverse PDE problems
J. Comput. Phys.
(2020)
Z. Mao et al.
Physics-informed neural networks for high-speed flows
Comput. Methods Appl. Math.
(2020)
L. Yan et al.
Adaptive multi-fidelity polynomial chaos approach to Bayesian inference in inverse problems
J. Comput. Phys.
(2019)
X. Luo et al.
Bayesian deep learning with hierarchical prior: predictions from limited and noisy data
Struct. Saf.
(2020)
G. Pang et al.
Neural-net-induced Gaussian process regression for function approximation and PDE solution
J. Comput. Phys.
(2019)
M. Raissi et al.
Inferring solutions of differential equations using noisy multi-fidelity data
J. Comput. Phys.
(2017)

Y. LeCun et al.

Deep learning

Nature

(2015)

S.H. Rudy et al.

Data-driven discovery of partial differential equations

Sci. Adv.

(2017)

N.M. Mangan et al.

Model selection for dynamical systems via sparse regression and information criteria

Proc. R. Soc. Lond. Ser. A, Math. Phys. Sci.

(2017)

S.L. Brunton et al.

Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control

(2019)

L. Lu et al.

DeepXDE: a deep learning library for solving differential equations

L. Yang et al.

Physics-informed generative adversarial networks for stochastic differential equations

SIAM J. Sci. Comput.

(2020)

X. Meng et al.

PPINN: parareal physics-informed neural network for time-dependent PDEs

J. Li et al.

Adaptive construction of surrogates for the Bayesian solution of inverse problems

SIAM J. Sci. Comput.

(2014)

L. Yan et al.

An adaptive surrogate modeling based on deep neural networks for large-scale Bayesian inverse problems

Cited by (580)

Inverse flow prediction using ensemble PINNs and uncertainty quantification
2024, International Journal of Heat and Mass Transfer
The thermal boundary conditions in a numerical simulation for heat transfer are often imprecise. This leads to poorly defined boundary conditions for the energy equation. The lack of accurate thermal boundary conditions in real-world applications makes it impossible to effectively solve the problem, regardless of the advancement of conventional numerical methods.
This study utilises a physics-informed neural network to tackle ill-posed problems for unknown thermal boundaries with limited sensor data. The network approximates velocity and temperature fields while adhering to the Navier-Stokes and energy equations, revealing unknown thermal boundaries and reconstructing the flow field around a square cylinder. Optimal sensor placement, determined by the QR pivoting technique, enhances the capture of dynamics, improving model accuracy. An ensemble PINN approach is implemented to increase robustness and generalisability, mitigating overfitting and underfitting risks, and providing a measure of model confidence. This enables the identification of reliable prediction regions and highlights potential inaccuracies, broadening applicability in complex heat transfer problems with unknown boundary conditions. Key findings include the ensemble physics-informed neural networks' superior predictive accuracy over single models and its ability to quantify uncertainty, offering insights into model validity. The study further highlights the importance of sensor placement, boundary condition enforcement, and activation function choice, with a notable shift from tanh to sin functions improving vortex shedding depiction.
A MCMC method based on surrogate model and Gaussian process parameterization for infinite Bayesian PDE inversion
2024, Journal of Computational Physics
This work focuses on an inversion problem derived from parametric partial differential equations (PDEs) with an infinite-dimensional parameter, represented as a coefficient function. The objective is to estimate this coefficient function accurately despite having only noisy measurements of the PDE solution at sparse input points. Conventional methods for inversion require numerous calls to a refined PDE solver, resulting in significant computational complexity, especially for challenging PDE problems, making the inference of the coefficient function practically infeasible. To address this issue, we propose a MCMC method that combines an deep learning-based surrogate and Gaussian process parameterization to efficiently infer the posterior of the coefficient function. The surrogate model is a combination of a cost-effective coarse PDE solver and a neural network-based transformation which provides an approximate solution derived from the refined PDE solver but based on the coarse PDE solution. The coefficient function is represented by Gaussian process with finite number of spatially dependent parameters and this parametric representation will be beneficial for the preconditioned Crank-Nicolson (pCN) Markov chain Monte Carlo method to sample efficiently from the posterior distribution. Approximate Bayesian computation method is used for constructing an informative dataset for the transformation learning. Our numerical examples demonstrate the effectiveness of this approach in accurately estimating the coefficient function while significantly reducing the computational burden associated with traditional inversion techniques.
Tensor-based physics-encoded neural networks for modeling constitutive behavior of soil
2024, Computers and Geotechnics
Data-driven constitutive models are increasingly addressing non-elastic and three-dimensional scenarios. However, their robustness can be significantly impacted by the inadequate integration of physical information. Accordingly, this study introduces a tensor-based physics-encoded neural network to characterize the constitutive behavior of soil, exemplified by isotropic hypoplasticity with dependency on pressure and porosity. The framework ensures strict adherence to fundamental physical laws for small strain, rate-independent isotropic constitutive behavior. The network utilizes stress tensor invariants and soil state parameters (porosity) as inputs, and outputs crucial coefficients for the tensorial constitutive relations. The model has been calibrated using only symmetric triaxial test data (both drained and undrained). The effectiveness of the neural network-based constitutive model has been validated through simulations of drained and undrained triaxial tests under various initial conditions, as well as boundary value problems with complex loading. The results demonstrate that the proposed model offers the following distinguishing advantages: 1) Applicability to both two- and three-dimensional non-elastic cases, even when trained with two-dimensional data; 2) Strict adherence to fundamental physical laws, avoiding soft constraints; 3) An incremental, tensor-based architecture, suitable for integration in numerical software for boundary value problems.
Physics-informed data-driven model of dehydration reaction stage in the sintering process of ternary cathode materials
2024, Control Engineering Practice
The sintering process is the main process that affects the performance of ternary cathode materials. Due to the closed, continuous, and long-term characteristics of the sintering process carried out in a roller kiln, the reaction state including reaction degree, particle size, and etc., of the raw materials cannot be directly obtained through sampling detection, resulting in ineffective temperature control. This article mainly focuses on modeling the dehydration reaction stage of the sintering process to obtain the reaction state of the raw materials. Firstly, a two-sphere model is established based on Newton’s second law and the particle volume conservation equation to describe the relationship between the particle radius and sintering temperature of lithium hydroxide and ternary precursor; Secondly, the chemical kinetic parameters of the dehydration reaction in the two-sphere model are obtained by combining the single heating rate scanning method with the multiple heating rate scanning method; Then, in order to improve the accuracy of the two-sphere model and reduce the impact of external environmental temperature changes, a physics-informed neural network model is constructed by combining the two-sphere model with the neural ordinary differential equation model. Finally, the effectiveness of the modeling method is verified through numerical simulation and empirical data.
Advancing wave equation analysis in dual-continuum systems: A partial learning approach with discrete empirical interpolation and deep neural networks
2024, Journal of Computational and Applied Mathematics
In this work, we propose a partial learning approach using partially explicit discretization for solving the wave equations. The considered mathematical model involves a dual-continuum system of hyperbolic second-order partial differential equations. We employ a finite element method for spatial approximation, while for temporal approximation, we utilize a partially explicit scheme. Specifically, we adopt the explicit scheme for the low-conductive continuum and the implicit scheme for the high-conductive continuum.
Our proposed method’s key idea lies in employing a partial learning approach to remove the necessity of finding the difficult-to-compute (implicit) part of the solution. Instead of attempting a full training of the numerical solution, we focus on training only a portion of it that requires the most computational cost. We use a proper orthogonal decomposition with a discrete empirical interpolation method to facilitate machine learning. By employing these methods, we can train a deep neural network to predict the values of the implicit part of the solution at specific observed locations and then restore the field.
We consider a two-dimensional model problem in heterogeneous media to test the effectiveness of the proposed approach. The numerical results demonstrate that our partial learning approach yields a satisfactory approximation. By combining the reduced-order modeling techniques, the partially explicit time scheme, and the deep neural network, we removed the necessity of finding the difficult-to-compute part of the solution while maintaining high accuracy.
Correcting model misspecification in physics-informed neural networks (PINNs)
2024, Journal of Computational Physics
Data-driven discovery of governing equations in computational science has emerged as a new paradigm for obtaining accurate physical models and as a possible alternative to theoretical derivations. The recently developed physics-informed neural networks (PINNs) have also been employed to learn governing equations given data across diverse scientific disciplines, e.g., in biology and fluid dynamics. Despite the effectiveness of PINNs for discovering governing equations, the physical models encoded in PINNs may be misspecified in complex systems as some of the physical processes may not be fully understood, leading to the poor accuracy of PINN predictions. In this work, we present a general approach to correct the misspecified physical models in PINNs for discovering governing equations, given some sparse and/or noisy data. Specifically, we first encode the assumed physical models, which may be misspecified in PINNs, and then employ other deep neural networks (DNNs) to model the discrepancy between the imperfect models and the observational data. Due to the expressivity of DNNs, the proposed method is capable of reducing the computational errors caused by the model misspecification and thus enables the applications of PINNs in complex systems where the physical processes are not exactly known. Furthermore, we utilize the Bayesian physics-informed neural networks (B-PINNs) and/or ensemble PINNs to quantify uncertainties arising from noisy and/or gappy data in the discovered governing equations. A series of numerical examples including reaction-diffusion systems and non-Newtonian channel and cavity flows demonstrate that the added DNNs are capable of correcting the model misspecification in PINNs and thus reduce the discrepancy between the physical models encoded in PINNs and the observational data. In addition, the B-PINNs and ensemble PINNs can provide reasonable uncertainty bounds in the discovered physical models, which makes the predictions more reliable. We also demonstrate that we can seamlessly combine the present approach with the symbolic regression to obtain the explicit governing equations upon the training of PINNs. We envision that the proposed approach will extend the applications of PINNs for discovering governing equations in problems where the physico-chemical or biological processes are not well understood.

View all citing articles on Scopus

¹: The first two authors contributed equally to this work.

View full text

B-PINNs: Bayesian physics-informed neural networks for forward and inverse PDE problems with noisy data

Abstract

Introduction

Section snippets

B-PINNs: Bayesian physics-informed neural networks

Results and discussion

Comparison with PINNs

Comparison with the truncated Karhunen-Loève expansion

Summary

CRediT authorship contribution statement

Declaration of Competing Interest

Acknowledgement

J. Comput. Phys.

J. Comput. Phys.

J. Comput. Phys.

J. Comput. Phys.

J. Comput. Phys.

Comput. Methods Appl. Math.

J. Comput. Phys.

Struct. Saf.

J. Comput. Phys.

J. Comput. Phys.

Deep learning

Nature

Data-driven discovery of partial differential equations

Sci. Adv.

Model selection for dynamical systems via sparse regression and information criteria

Proc. R. Soc. Lond. Ser. A, Math. Phys. Sci.

Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control

DeepXDE: a deep learning library for solving differential equations

Physics-informed generative adversarial networks for stochastic differential equations

SIAM J. Sci. Comput.

PPINN: parareal physics-informed neural network for time-dependent PDEs

Adaptive construction of surrogates for the Bayesian solution of inverse problems

SIAM J. Sci. Comput.

An adaptive surrogate modeling based on deep neural networks for large-scale Bayesian inverse problems