Markov chain analysis of evolutionary algorithms on OneMax function – From coupon collector's problem to (1 + 1) EA

Abstract

The theoretical investigation of Evolutionary Algorithms (EAs) has increased our understanding of the computational mechanism of algorithms. OneMax is a test function most frequently and deeply studied in the field of EAs. In this work, a method is presented for describing the runtime properties of $(1+1)$ EA on OneMax. This method is motivated by the work of Erdös and Rényi treating the coupon collector's problem. They showed that the success probability of the coupon collector's problem is given by a function of double exponential form, and that the number of uncollected coupons follows the Poisson distribution. Today, the double exponential function is called Gumbel function, which is one of three fundamental functions in extreme value statistics. We introduce an algorithm that is a variant of the $(1+1)$ EA, First Order Evolutionary Algorithm (FO-EA). FO-EA takes into account only the effect of single-bit mutations in the $(1+1)$ EA, which in general includes multiple-bit mutations. We modified the method of Erdös and Rényi to apply FO-EA. We apply the Gumbel distribution for calculating the success probability of the $(1+1)$ EA on OneMax. This method turns out to give a sufficiently reliable estimation for success probabilities, even in the tail region.

Introduction

The theoretical study of randomized search heuristics is now a growing research domain, and many interesting results have been obtained [1]. This class of approaches contains Randomized Local Search (RLS) and Evolutionary Algorithms (EAs). EAs have been investigated by many researchers whose main interests are their convergence properties. Especially, computational time analysis is a major challenge and hot spot in the study of the basic theory of EAs [2].

The computational time of an EA refers to the number of iterations required by the algorithm to find the optimal solution, also known as runtime, which is an important index to measure the algorithm's advantages and disadvantages. If T is a random variable measuring the time to obtain the optimum solution, the main task of runtime analysis is estimating $\mathrm{E}(T)$, the expected value of T. For the convenience of analysis, many studies have been conducted on the $(1+1)$ EA. Although the $(1+1)$ EA is a very simple form of EA, it has a value of theoretical analysis and is a basic model for runtime analysis. By studying the $(1+1)$ EA deeply, we can reveal the dynamic behavior of EAs and provide a useful tool for analyzing more complex algorithms. A detailed overview of the $(1+1)$ EAs is presented in [3].

Droste et al. [4] carried out a rigorous probabilistic analysis of the $(1+1)$ EA for pseudo-Boolean functions such as OneMax, LeadingOnes and BinVal functions. They proved that the average running time of the $(1+1)$ EA for solving linear functions is $\mathrm{\Theta }(\ell \log \ell )$, where ℓ is the length of binary string. He and Yao proposed another type of mathematical model, usually called as drift analysis, for the study of EAs [5], [6]. Drift analysis is based on the drift theorem, which was introduced as a widely applicable mathematical model for the computation time analysis of EAs. Later, a new drift theorem (multiplicative drift theorem) was introduced by Doerr et al. [7], [8]. Multiplicative drift analysis allows easier analyses if the optimization progress is roughly proportional to the current distance to the optimum. Hwan et al. performed their probabilistic analysis on the runtime of the $(1+1)$ EA for OneMax problem [9]. They reported a new asymptotic approximation for the expected hitting time m with $p_m=1/\ell$. The result is given by

$$m=e\ell \log \ell +c_1\ell +\frac{1}{2}e\log \ell +c_2+O({\ell }^{-1}\log \ell ),$$

where $c_1\approx 1.8925$ and $c_2\approx 0.5979$.

A framework of Markov chains for analyzing the average hitting times of EAs was presented He and Yao [10]. The behavior of this Markov chain is completely determined by the transition matrix P of an absorbing Markov chain [11]. Mülenbein carried out a pioneering study by using Markov chain for analyzing the time evolution of states in the $(1+1)$ EA [12]. He derived the expected hitting time m of the optimum of OneMax function. Jägersküpper [13] combined the Markov chain with drift analysis and reanalyzed the upper bound of running time of the $(1+1)$ EA on the linear function to obtain a more accurate estimate. These results show that the model making full use of Markov property of EA is a powerful tool for analyzing the dynamical behavior of EA. Furutani et al. also analyzed the expected hitting time of the $(1+1)$ EAs by applying Markov chain, and the explicit form of absorbing Markov chain for OneMax problem was presented [14].

Most of above mentioned studies treat the mean and median of the runtime distribution. There is another important performance measure, the runtime distribution in the tail region of large t. The study of tail distribution of runtime provides useful information that can contribute to the design of better algorithms. However, only a small number of studies have investigated the tail distribution in evolutionary algorithms. There is a branch in statistics and probability theory, which deals with the distribution of (exceptionary) large or small values. This field, usually called extreme value statistics, has been applied to problems of natural disaster, insurance, engineering, and so on. There are three types in the extreme value distribution, and the Gumbel distribution is one of them [15]. In ref. [9], it is proven that the success probability of OneMax function asymptotically approaches the Gumbel distribution.

The main objective of this paper is to study the success probability for the $(1+1)$ EA on OneMax function. The success probability is defined as $F(t)=\mathrm{Pr}(T\le t)$ given a certain number of steps t. For this end, we take into account of theoretical achievements in the study of coupon collector's problem. As Doerr stated in his paper [16], “understanding its solution pays off when analyzing randomized search heuristics, because very similar situations are encountered here”. Especially, we were interested in the results of Erdős and Rényi [17]. They treated the urn problem, which is equivalent to the coupon collector's problem. Their main result is summarized;

• the success probability of the coupon collector's problem is given by a function of double exponential type;

• the number of uncollected coupons obeys the Poisson distribution.

In this paper, we demonstrate that these two findings can be applicable to the $(1+1)$ EA on OneMax function in the case of large enough ℓ.

For an intermediate step to the goal, we formalize the Markov chain model for RLS on OneMax function. Since the transition matrix of RLS chain has a triangular form [3], its diagonal elements are also eigenvalues of the matrix. Their corresponding eigenvectors are given in terms of binomial coefficients. These results can be used to verify the findings of Erdős and Rényi.

As a simplified model for the application of Markov chain, we introduce a modified form of the $(1+1)$ EA, named as FO-EA short for First Order EA. FO-EA is designed as a generalization of RLS for the mathematical analysis of the $(1+1)$ EA on OneMax. The transition matrix for the Markov chain model of FO-EA only contains the first order terms, which are proportional to $p_m{(1-p_m)}^{\ell -1}$. This makes the Markov chain analysis tractable.