Analyzing randomness effects on the reliability of exploratory landscape analysis

Abstract

The inherent difficulty of solving a continuous, static, bound-constrained and single-objective black-box optimization problem depends on the characteristics of the problem’s fitness landscape and the algorithm being used. Exploratory landscape analysis (ELA) uses numerical features generated via a sampling process of the search space to describe such characteristics. Despite their success in a number of applications, these features have limitations related with the computational costs associated with generating accurate results. Consequently, only approximations are available in practice which may be unreliable, leading to systemic errors. The overarching aim of this paper is to evaluate the reliability of five well-known ELA feature sets across multiple dimensions and sample sizes. For this purpose, we propose a comprehensive experimental methodology combining exploratory and statistical validation stages, which uses resampling techniques to minimize the sampling cost, and statistical significance tests to identify strengths and weaknesses of individual features. The data resulting from the methodology is collected and made available in the LEarning and OPtimization Archive of Research Data v1.0. The results show that instances of the same function can have feature values that are significantly different; hence, non-generalizable across instances, due to the effects produced by the boundary constraints. In addition, some landscape features under evaluation are highly volatile, and strongly susceptible to changes in sample size. Finally, the results show evidence of a curse of modality, meaning that the sample size should increase with the number of local optima.

Appendix: Validation of the assumptions behind the experimental methodology

Our experimental methodology makes assumptions that can be summarized in two questions: (a) Since Latin Hyper-cube Samplin (LHS) is a type of stratified sampling, is the independence assumption still valid? (b) What are the differences between multiple uniformly distributed random samples, bootstrapping a single uniformly distributed random sample, and bootstrapping a single LHS, when calculating the variance of an estimate? To answer these questions, we have carried out two simple experiments that demonstrate that there is no practical difference on the results between taking multiple uniformly distributed random samples and bootstrapping a LHS. On the first experiment, we address the independence assumption, by calculating the magnitude of the auto-correlation with lags in the $\left[ 1,\ 50\right]$ range, for data drawn from the $\left[ 0,\ 1\right]$ interval. For this assumption to hold for LHS, the magnitudes of the auto-correlation should follow the same trend that for a uniformly distributed random sample and be close to zero, indicating that it is not possible to estimate the value of one point from another. We repeat this experiment 1000 times and average the results, which are presented in Fig. 16a for samples with $\left\{ 200,600,1000\right\}$ points. Other than the descending trend for a sample of 200 points, which can be explained by the decrease in points in the sample for which the auto-correlation can be calculated, the results demonstrate that the independence assumption holds for a LHS in practice.

Fig. 16

Validation of the assumptions behind our experimental methodology. a Average magnitude of the auto-correlation with lags in the $\left[ 1,\ 50\right]$ range, for data drawn from the $\left[ 0,\ 1\right]$ interval. b Distribution of the variance of the mean from N samples of n points, $IID\left( n,N\right)$, bootstrapping N times a sample of n points, $IID+B\left( n,N\right)$, and bootstrapping N times a LHS of n points, $LHS+B\left( n,N\right)$. The results confirm that there is no practical difference between taking N uniformly distributed random samples and bootstrapping N times a single LHS

On the second experiment, we address the second question by estimating the variance of the mean from these three different sampling regimes, using data drawn from the $\left[ 0,\ 1\right]$ interval. On the first one, called $IID\left( n,N\right)$, we took N uniformly distributed random samples of n points. On the second one, called $IID+B\left( n,N\right)$, we took one uniformly distributed sample of n points and bootstrapped it N times. On the third one, called $LHS+B\left( n,N\right)$, we took one LHS of n points and bootstrapped it N times. Each sampling regime produced N mean estimates, from which the variance is calculated. The experiments are repeated 1000 times for all the combinations of $\left\{ n,N\right\} =\left\{ 200,600,1000\right\}$. The results are shown in Fig. 16b as box-plots, which demonstrate that there is no practical difference between taking N uniformly distributed random samples and bootstrapping N times a single LHS.