Skip to main content
SpringerLink
Log in

Adaptive design of experiments for global Kriging metamodeling through cross-validation information

  • Research Paper
  • Published:
Structural and Multidisciplinary Optimization Aims and scope Submit manuscript

Abstract

This paper discusses a new sequential adaptive design of experiments (DoE) approach for global Kriging metamodeling applications. The sequential implementation is established by using the current metamodel, formulated based on the existing experiments, to guide the selection of the optimal new experiment(s). The score function, defining the DoE objective, combines two components: (1) the metamodel prediction variability, expressed through the predictive variance, and (2) the metamodel bias, approximated through the leave-one-out cross validation (LOOCV) error. The latter is used as a weighting factor to extend traditional DoE approaches that focus solely on the metamodel prediction variability. Two such approaches are considered here, adopting either the integrated mean squared error or the maximum mean squared error as the basic component of the score function. The incorporation of bias information as weighting within these well-established approaches facilitates a direct extension of their respective computational workflows, making the proposed implementation attractive from computational perspective. An efficient optimization scheme for identification of the next experiment, as well as the balancing of exploration and exploitation between the two components of the score function, are also discussed. The incorporation of LOOCV weightings is shown to be highly beneficial in a total of six analytical and engineering examples. Furthermore, these examples demonstrate that for DoE approaches which use LOOCV information as weights, it is preferable to update the predictive variance to explicitly consider the impact of the new experiment, rather than relying strictly on the current metamodel variance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  • Aute V, Saleh K, Abdelaziz O, Azarm S, Radermacher R (2013) Cross-validation based single response adaptive design of experiments for Kriging metamodeling of deterministic computer simulations. Struct Multidiscip Optim 48(3):581–605

    Article  Google Scholar 

  • Bect J, Li L, Vazquez E (2017) Bayesian subset simulation. SIAM/ASA Journal on Uncertainty Quantification 5(1):762–786

    Article  MathSciNet  Google Scholar 

  • Chen T, Morris J, Martin E (2007) Gaussian process regression for multivariate spectroscopic calibration. Chemom Intell Lab Syst 87(1):59–71

    Article  Google Scholar 

  • Dixon LCW, Szegö GP (1978) Towards global optimisation, vol 2. North-Holland Amsterdam,

  • Dubourg V, Sudret B, Bourinet J-M (2011) Reliability-based design optimization using kriging surrogates and subset simulation. Struct Multidiscip Optim 44(5):673–690

    Article  Google Scholar 

  • Dubrule O (1983) Cross validation of kriging in a unique neighborhood. J Int Assoc Math Geol 15(6):687–699

    Article  MathSciNet  Google Scholar 

  • Falnes J (2002) Ocean waves and oscillating systems: linear interactions including wave-energy extraction. Cambridge university press,

  • Forrester AI, Keane AJ (2009) Recent advances in surrogate-based optimization. Prog Aerosp Sci 45(1–3):50–79

    Article  Google Scholar 

  • Hartigan JA, Wong MA (1979) Algorithm AS 136: a K-means clustering algorithm. J R Stat Soc: Ser C: Appl Stat 28(1):100–108

    MATH  Google Scholar 

  • Hu Z, Mahadevan S (2016) A single-loop kriging surrogate modeling for time-dependent reliability analysis. J Mech Design 138(6):061406

    Article  Google Scholar 

  • Jia G, Taflanidis AA (2013) Kriging metamodeling for approximation of high-dimensional wave and surge responses in real-time storm/hurricane risk assessment. Comput Methods Appl Mech Eng 261:24–38

    Article  MathSciNet  Google Scholar 

  • Jia G, Taflanidis AA (2014) Sample-based evaluation of global probabilistic sensitivity measures. Comput Struct 144:103–118

    Article  Google Scholar 

  • Jiang P, Shu L, Zhou Q, Zhou H, Shao X, Xu J (2015) A novel sequential exploration-exploitation sampling strategy for global metamodeling. IFAC-PapersOnLine 48(28):532–537

    Article  Google Scholar 

  • Jin R, Chen W, Sudjianto A (2002) On sequential sampling for global metamodeling in engineering design. In: ASME 2002 international design engineering technical conferences and computers and information in engineering conference. American Society of Mechanical Engineers, pp 539-548

  • Johnson ME, Moore LM, Ylvisaker D (1990) Minimax and maximin distance designs. J Stat Plan Infer 26(2):131–148

    Article  MathSciNet  Google Scholar 

  • Jones DR, Schonlau M, Welch WJ (1998) Efficient global optimization of expensive black-box functions. J Glob Optim 13(4):455–492

    Article  MathSciNet  Google Scholar 

  • Kennedy MC, O'Hagan A (2001) Bayesian calibration of computer models. J R Stat Soc Series B Stat Methodol 63(3):425–464

    Article  MathSciNet  Google Scholar 

  • Kianifar MR, Campean F (2019) Performance evaluation of metamodelling methods for engineering problems: towards a practitioner guide. Struct Multidiscip Optim:1–28

  • Kleijnen JP (2009) Kriging metamodeling in simulation: a review. Eur J Oper Res 192(3):707–716

    Article  MathSciNet  Google Scholar 

  • Kleijnen JP (2015) Design and analysis of simulation experiments. International Workshop on Simulation. Springer, In, pp 3–22

    Book  Google Scholar 

  • Kleijnen JP, Beers W (2004) Application-driven sequential designs for simulation experiments: Kriging metamodelling. J Oper Res Soc 55(8):876–883

    Article  Google Scholar 

  • Koehler J, Owen A (1996) Computer experiments. Handbook of statistics. (S. Ghosh and CR Rao, eds). Elsevier science, New York

  • Le Gratiet L, Cannamela C (2015) CoKriging-based sequential design strategies using fast cross-validation techniques for multi-fidelity computer codes. Technometrics 57(3):418–427

    Article  MathSciNet  Google Scholar 

  • Liu H, Ong Y-S, Cai J (2018) A survey of adaptive sampling for global metamodeling in support of simulation-based complex engineering design. Struct Multidiscip Optim 57(1):393–416

    Article  Google Scholar 

  • Liu H, Xu S, Ma Y, Chen X, Wang X (2016) An adaptive Bayesian sequential sampling approach for global metamodeling. J Mech Design 138(1):011404

    Article  Google Scholar 

  • Lophaven SN, Nielsen HB, Søndergaard J (2002) DACE-A Matlab Kriging toolbox, version 2.0

  • Mavroeidis GP, Papageorgiou AS (2003) A mathematical representation of near-fault ground motions. Bull Seismol Soc Am 93(3):1099–1131

    Article  Google Scholar 

  • McKay MD, Beckman RJ, Conover WJ (1979) Comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2):239–245

    MathSciNet  MATH  Google Scholar 

  • Mehmani A, Chowdhury S, Meinrenken C, Messac A (2018) Concurrent surrogate model selection (COSMOS): optimizing model type, kernel function, and hyper-parameters. Struct Multidiscip Optim 57(3):1093–1114

    Article  MathSciNet  Google Scholar 

  • Mishra SK (2006) Global optimization by differential evolution and particle swarm methods: evaluation on some benchmark functions. Available at SSRN 933827

  • Picheny V, Ginsbourger D, Roustant O, Haftka RT, Kim N-H (2010) Adaptive designs of experiments for accurate approximation of a target region. J Mech Design 132(7):071008

    Article  Google Scholar 

  • Picheny V, Wagner T, Ginsbourger D (2013) A benchmark of kriging-based infill criteria for noisy optimization. Struct Multidiscip Optim 48(3):607–626

    Article  Google Scholar 

  • Queipo NV, Haftka RT, Shyy W, Goel T, Vaidyanathan R, Tucker PK (2005) Surrogate-based analysis and optimization. Prog Aerosp Sci 41(1):1–28

    Article  Google Scholar 

  • Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Stat Sci 4(4):409–435

    Article  MathSciNet  Google Scholar 

  • Santner TJ, Williams BJ, Notz W, Williams BJ (2003) The design and analysis of computer experiments, vol 1. Springer

  • Schöbi R, Sudret B, Marelli S (2016) Rare event estimation using polynomial-chaos kriging. ASCE-ASME J. Risk Uncertain. Eng. Syst., part a. Civ Eng 3(2):D4016002

  • Shewry MC, Wynn HP (1987) Maximum entropy sampling. J Appl Stat 14(2):165–170

    Article  Google Scholar 

  • Simpson TW, Mauery TM, Korte JJ, Mistree F (2001a) Kriging models for global approximation in simulation-based multidisciplinary design optimization. AIAA J 39(12):2233–2241

    Article  Google Scholar 

  • Simpson TW, Poplinski J, Koch PN, Allen JK (2001b) Metamodels for computer-based engineering design: survey and recommendations. Eng Comput 17(2):129–150

    Article  Google Scholar 

  • Spall JC (2005) Introduction to stochastic search and optimization: estimation, simulation, and control, vol 65. John Wiley & Sons

  • Sundararajan S, Keerthi SS (2001) Predictive approaches for choosing hyperparameters in Gaussian processes. Neural Comput 13(5):1103–1118

    Article  Google Scholar 

  • Van Beers WC, Kleijnen JP (2008) Customized sequential designs for random simulation experiments: Kriging metamodeling and bootstrapping. Eur J Oper Res 186(3):1099–1113

    Article  Google Scholar 

  • Williams CK, Rasmussen CE (2006) Gaussian processes for machine learning, vol 2. vol 3. MIT Press Cambridge, MA

    MATH  Google Scholar 

  • Zhang J, Taflanidis AA, Nadal-Caraballo NC, Melby JA, Diop F (2018) Advances in surrogate modeling for storm surge prediction: storm selection and addressing characteristics related to climate change. Nat Hazards 94(3):1225–1253

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Civil & Environmental Engineering & Earth Sciences, University of Notre Dame, Notre Dame, IN, 46556, USA

    Aikaterini P. Kyprioti, Jize Zhang & Alexandros A. Taflanidis

  2. Department of Aerospace and Mechanical Engineering, University of Notre Dame, Notre Dame, IN, 46556, USA

    Alexandros A. Taflanidis

Authors
  1. Aikaterini P. Kyprioti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jize Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alexandros A. Taflanidis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexandros A. Taflanidis.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest..

Replication of results

Numerical simulations were performed in MATLAB 2017b. Calibration of the surrogate model hyper-parameters was performed utilizing DACE toolbox (Lophaven et al. 2002). Numerical codes for the DoE optimization, described in detail in Section 3.2 of the manuscript, can be obtained from the corresponding author with reasonable request.

Additional information

Responsible Editor: Shapour Azarm

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kyprioti, A.P., Zhang, J. & Taflanidis, A.A. Adaptive design of experiments for global Kriging metamodeling through cross-validation information. Struct Multidisc Optim 62, 1135–1157 (2020). https://doi.org/10.1007/s00158-020-02543-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00158-020-02543-1

Keywords

Search

Navigation