Bayesian optimization using deep Gaussian processes with applications to aerospace system design

Abstract

Bayesian Optimization using Gaussian Processes is a popular approach to deal with optimization involving expensive black-box functions. However, because of the assumption on the stationarity of the covariance function defined in classic Gaussian Processes, this method may not be adapted for non-stationary functions involved in the optimization problem. To overcome this issue, Deep Gaussian Processes can be used as surrogate models instead of classic Gaussian Processes. This modeling technique increases the power of representation to capture the non-stationarity by considering a functional composition of stationary Gaussian Processes, providing a multiple layer structure. This paper investigates the application of Deep Gaussian Processes within Bayesian Optimization context. The specificities of this optimization method are discussed and highlighted with academic test cases. The performance of Bayesian Optimization with Deep Gaussian Processes is assessed on analytical test cases and aerospace design optimization problems and compared to the state-of-the-art stationary and non-stationary Bayesian Optimization approaches.

Appendices

Appendix A: Functions

Modified Xiong function (Fig. 22):

$$\begin{aligned} f(x)=-0.5\left( \sin \left( 40(x-0.85)^4\right) \cos \left( 2.5(x-0.95)\right) +0.5(x-0.9)+1\right) , \text { } x \in [0,1] \end{aligned}$$

(18)

Modified TNK constraint function (Fig. 23):

$$\begin{aligned} f({\mathbf{x }})=1.6(x_0-0.6)^2+1.6(x_1-0.6)^2-0.2\cos \left( 20\arctan \left( \frac{0.3x_0}{(x_1+10^{-8})}\right) \right) -0.4, \text { } {\mathbf{x }}\in [0,1]\times [0,1] \end{aligned}$$

(19)

Fig. 22

Modified Xiong function

Fig. 23

Modified TNK constraint

10d Trid function (Fig. 24):

$$\begin{aligned} f({\mathbf{x }})=\sum _{i=1}^{10} (x_i-1)^2-\sum _{i=2}^{10}x_ix_{i-1}, \text { } x_i \in [-100,100], \forall i=1,\dots ,10 \end{aligned}$$

(20)

Hartmann-6d function (Fig. 25):

$$\begin{aligned} f({\mathbf{x }})= \sum _{i=1}^4 \alpha _i \exp \left( -\sum _{j=1}^6 A_{ij} (x_j-Pij)^2 \right) , \text { } x_i\in [0,1], \forall i=1,\dots ,6 \end{aligned}$$

(21)

with:

$$\begin{aligned} \alpha =[1,1.2,3,3.2]^\top \end{aligned}$$

and

$$\begin{aligned} P=10^{-4} \begin{bmatrix} 1312 &{} 1696 &{} 5569 &{} 124 &{} 8283 &{} 5886\\ 2329 &{} 4135 &{} 8307&{}3736&{}1004&{}9991\\ 2348&{}1451&{}3522&{}2883&{}3047&{}6650\\ 4047&{}8828&{}8732&{}5743&{}1091&{}381\end{bmatrix} \end{aligned}$$

and

$$\begin{aligned} A= \begin{bmatrix} 10&{}3&{}17&{}3.5&{}1.7&{}8\\ 0.05&{}10&{}17&{}0.1&{}8&{}14\\ 3&{}3.5&{}1.7&{}10&{}17&{}8\\ 17&{}8&{}0.05&{}10&{}0.1&{}14 \end{bmatrix} \end{aligned}$$

Fig. 24

Sectional 2d view of the Trid function showing where the global minimum lies

Fig. 25

Sectional 2d view of the Hartmann-6d function showing where the global minimum lies

Appendix B: Experimental setup

• All experiments were executed on Grid’5000 using a Tesla P100 GPU. The code is based on GPflow (de G Matthews et al. 2017) and Doubly-Stochastic-DGP (Salimbeni and Deisenroth 2017).

• For all DGPs, RBF kernels are used with a length-scale and variance initialized to 1 if it does not get an initialization from a previous DGP. The data is scaled to have a zero mean and a variance equal to 1.

• The Adam optimizer is set with $\beta _1=0.8$ and $\beta _2=0.9$ and a step size $\gamma ^{adam}=0.01$.

• The natural gradient step size is initialized for all layers at $\gamma ^{nat}=0.1$

• For BO with DGP the number of successive updates before optimizing from scratch is 5.

• The infill criteria are optimized using a parallel differential evolution algorithm with a population of 400 and 100 generations.

• A Github repository featuring BO & DGP algorithm will be available after the publication of the paper.