Using a Gaussian process regression inspired method to measure agreement between the experiment and CFD simulations

Highlights

• The Gaussian process regression(GPR) model is used to fit the experimental data.

• The agreement between an experiment and the simulation is replaced by comparing outputs of the simulation and the GPR model.

• Two metrics are used to provide tangible information for the local and global agreement, repectively.

• The quantitative information helps to make an objective argument for the accuracy level of a CFD model.

Abstract

This paper presents a Gaussian process regression inspired method to measure the agreement between experiment and computational fluid dynamics (CFD) simulation. Because of misalignments between experimental and numerical outputs in spatial or parameter space, experimental data are not always suitable for quantitative assessing the numerical models. In this proposed method, the cross-validated Gaussian process regression (GPR) model, trained based on experimental measurements, is used to mimic the measurements at positions where there are no experimental data. The agreement between an experiment and the simulation is mimicked by the agreement between the simulation and GPR models. The statistically weighted square error is used to provide tangible information for the local agreement. The standardised Euclidean distance is used for assessing the overall agreement.

The method is then used to assess the performance of four scale-resolving CFD methods, such as URANS k-ω-SST, SAS-SST, SAS-KE, and IDDES-SST, in simulating a prism bluff-body flow. The local statistically weighted square error together with standardised Euclidean distance provide additional insight, over and above the qualitative graphical comparisons. In this example scenario, the SAS-SST model marginally outperformed the IDDES-SST and better than the other two other, according to the distance to the validated GPR models.

Introduction

Understanding complex flow phenomena is an important job in many in many fields, e.g., performance of electric gadgets (Conficoni et al., 2015; Kaplan et al., 2013), flow structure interaction (Glück et al., 2001; Hou et al., 2012), thermal hydraulics in complex system (Amin et al., 2018; Duan et al., 2014; Duan and He, 2017a, 2017b), as well as dispersion of air pollutants (Al-Abidi et al., 2013; Allegrini et al., 2014; Gilani et al., 2016; Gromke et al., 2015). Thanks to the rapid development of commercial computational fluid dynamics (CFD) software in conjunction with growing computing power, the replacement of full-scale experiments with sophisticated CFD models is a growing trend. However, high-fidelity CFD methods, such as DNS and LES, are still impractical in most industrial cases. Currently, the most widely used CFD methods, such as RANS and Hybrid LES/RANS, are subject to various levels of simplification and assumption, which introduce model bias into the predictions. Therefore, identifying the suitability of a CFD model for specific scenarios is an ongoing endeavour.

Graphical comparison is a common approach to assess the performance of CFD models or validation and calibration of turbulence models, for instance in (Billard et al., 2012; Duan et al., 2019, 2018a, 2018b; Keshmiri et al., 2016b, 2016a; Launder and Spalding, 1974; Menter, 2009; Revell et al., 2006; Shih et al., 1995). The approach qualitatively determines the agreement between the model and experiments by observing the data plotted in figures. By its nature, the conclusion of a graphical comparison is observer dependent and may be biased, especially when the results of different models are close to each other. Therefore, quantitative assessment of the agreement between CFD models and measurements is important in terms of providing a more objective conclusion for ranking the models and validation. The widely accepted goal of validation is the determination of the degree to which a model is an accurate representation of the physics from the perspective of intended uses of the model (AIAA, 1998; ASME, 2006; Oberkampf and Trucano, 2008; Oberkampf et al., 2002).

A review of previously developed validation methods can be found in the article (Lee et al., 2016; Ling and Mahadevan, 2013). The methods for assessing the validity of a numerical model can be classified into two types, namely hypothesis tests and metrics assessment. The hypothesis test aims to accept/reject the model, whilst metrics assessments focus on quantifying the agreement between the model and experiment. The hypothesis test, such as the p-value test and the Bayesian factor, were developed by considering either statistics of the experiment or the numerical simulation. Meanwhile, many metric assessment methods were developed and assessed, such as Euclidean distance (Audouin et al., 2011; Peacock et al., 1999), Mahalanobis distance (Rebba and Mahadevan, 2006; Zhao et al., 2017), the confidence interval (Barone et al., 2006; Oberkampf and Barone, 2006), and area metrics (Ferson et al., 2008; Ferson and Oberkampf, 2009). Both Euclidean distance and Mahalanobis distance measure the difference between two vectors. The former treats the squared error of each element in the vector equally, while the latter considers the bias due to the statistical features of the elements. The confidence interval is designed to provide the confidence level of the model error. Area metrics assess the similarity of the cumulative distribution of different stochastic processes obtained using experiments and simulations.

Additionally, a high fidelity database is required to validate the numerical method or assess the capability of a physical model. Instead of building a new test rig or starting a high-fidelity simulation, such as a direct numerical simulation (DNS), (which may not be practical) utilizing historical experimental databases is a more feasible and economical choice. In most cases, it is impossible to use measurements directly in a quantitative validation, because of the misalignment between the measurements and numerical outputs in the spatial or parameter domain. Barone et al. (2006), as well as, Oberkampf and Barone (2006) suggested to first fit a non-linear regression model to the measurements and then use the fitted curve to replace the experimental measurement. Accordingly, the confidence interval on the model error as well as the global agreement metric can be calculated even when the measures are sparse over the range of input parameter. One of the major limitations of this approach comes from the regression method. It is known that the functional form chosen for the non-linear regression will have a large impact on the results. Based on the Bayesian calibration procedure developed by Kennedy and O'Hagan (2001). Wang et al. (2009) suggested a framework using Gaussian process regression (GPR) to fit the curve as well as the confidence interval for the model error . The procedure also provides the mean and confidence interval on the model bias (error) over the observation domain. As a further development of Wang et al. (2009), Chen et al. (2008) developed a metric based on the p-value test designed to determine the accuracy of a model for design purposes. These existing methods are not suited to ranking the performance of different numerical models.

This work focuses on quantitatively assessing the difference between experiment and simulations, as well as ranking the numerical models used. It consists of two steps: curve-fitting of the observations using GPR and evaluation of the distance between the fitted curve and numerical outputs using metrics. The statistically weighted squared error is used to represent the local distance, whilst the standardised Euclidean distance is used to provide the overall distance between the model outputs and the fitted curve. The numerical models will be ranked in terms of performance by the comparison of their standardised Euclidean distances.

In current practice, CFD models are treated as deterministic. However, the numerical model is affected by many uncertainties arising from inadequate knowledge of physical system, e.g., variability in physical properties (Rebba and Mahadevan, 2008). Exploring the stochastic feature of a variable, using a numerical method, requires hundreds or even thousands of simulations. However, CFD simulations are often computationally expensive, especially for scale-resolved simulations. As a result, it is often too expensive to recreate the probability distributions of a variable purely using CFD simulations. It is possible to simulate the stochastic feature using a surrogate model to mimic the CFD model in the event domain. This will be pursued in our future work.

The rest of this paper is organised as follows: a brief introduction to the GPR method is given in Section 2. The definition of the statistically weighted squared error and the standardised Euclidean distances are listed in Section 3. Section 4 contributes to describing the experiment by Volvo (Sjunnesson et al., 1992) and CFD models. GPR predictions are validated in Section 5 before demonstrating the quantification procedure in Section 6. General conclusions are given in Section 7.