Augmented intuition: a bridge between theory and practice

Abstract

Motivated by the celebrated paper of Hooker (J Heuristics 1(1): 33–42, 1995) published in the first issue of this journal, and by the relative lack of progress of both approximation algorithms and fixed-parameter algorithms for the classical decision and optimization problems related to covering edges by vertices, we aimed at developing an approach centered in augmenting our intuition about what is indeed needed. We present a case study of a novel design methodology by which algorithm weaknesses will be identified by computer-based and fixed-parameter tractable algorithmic challenges on their performance. Comprehensive benchmarkings on all instances of small size then become an integral part of the design process. Subsequent analyses of cases where human intuition “fails”, supported by computational testing, will then lead to the development of new methods by avoiding the traps of relying only on human perspicacity and ultimately will improve the quality of the results. Consequently, the computer-aided design process is seen as a tool to augment human intuition. It aims at accelerating and foster theory development in areas such as graph theory and combinatorial optimization since some safe reduction rules for pre-processing can be mathematically proved via theorems. This approach can also lead to the generation of new interesting heuristics. We test our ideas with a fundamental problem in graph theory that has attracted the attention of many researchers over decades, but for which seems it seems to be that a certain stagnation has occurred. The lessons learned are certainly beneficial, suggesting that we can bridge the increasing gap between theory and practice by a more concerted approach that would fuel human imagination from a data-driven discovery perspective.

Appendices

The 10 reduction rules used to test the kernelization performance on benchmark instances

In parameterized complexity, for a problem that is in class FPT, it is possible to obtain, under some circumstances and after “thinking really hard”, an algorithm for which a tight upper bound on the complexity can be found.

This algorithm is obtained after the combined effort of the repeated application of the ten reduction rules for k -Vertex Cover which concentrate on different properties of the graph. We have implemented all of them and we used the reduction rules one after another on the output from the previously applied reduction rule. We will apply these rules repeatedly until no more reduction possible. We will call the resulting method The 10-R and we will analyze their performance on some sets of instances.

The ten reduction rules of the The 10-R are:

• Degree Zero

• Degree One

• Degree k

• Degree Two Adjacent

• Degree Two Non-adjacent

• Complete neighborhood

• Crown

• Linear-Programming (LP)

• Struction

• General Fold

In this section we will describe those reduction rules for a graph $G=(V,E)$ where we have $n=|V|$ and $m=|E|$. It is then useful to explicitly define them here as follows:

Reduction Rule 1

(Degree Zero) If G contains a vertex u such that $|N(u)|=0$, then remove u.

An isolated vertex is not incident on any edge. Hence, it can not be in a optimal vertex cover. We can eliminate a degree zero vertices form G and reduce the problem size n by one, but the parameter size remains at k.

Reduction Rule 2

(Degree One Balasubramanian et al. (1998)) If G contains a vertex u such that $N(u)=\{v\}$, then add v to the vertex cover and remove u and v from the graph.

A vertex with single degree can be removed form the graph if and only if its unique neighbor is included in the vertex cover. Removing u from G and adding v to the vertex cover will reduce size of the instance n by two and parameter to $k-1$.

Reduction Rule 3

(Degree k Buss and Goldsmith (1993)) If G contains a vertex u with $|N(u)|\ge k$, then add u to the vertex cover and remove it from the graph.

Selecting u to the vertex cover of the graph reduces the problem size to $n-1$ and the parameter size to $k-1$.

Reduction Rule 4

(Degree Two Adjacent Balasubramanian et al. (1998)) If G contains a vertex u such that $N(u)= \{v,w\}$ and $(vw) \in E$, then add v and w to the vertex cover and remove u, v and w from the graph.

At least two of three vertices u, v, w of a triangle should be selected to the vertex cover, and any optimal solution that does not choose v and w can be changed to one that does. It will reduce the problem size to $n-3$ and the parameter size reduces to $k-2$.

Reduction Rule 5

(Degree Two Non-adjacent Balasubramanian et al. (1998)) If G contains a vertex u such that $N(u)= \{v,w\}$ and $(vw) \notin E$, then we can fold u by contracting edges uv and uw. We can achieve this by replacing u, v, w with a new vertex $u'$, where $N(u') = N(v) \cup N(w)$.

Either u is chosen to cover the edges (uv) and (uw) and neither of v or w are chosen, or at least one of v or w is chosen, in which case it is sufficient to choose both and not u to be in the vertex cover Chen et al. (2001). If the new vertex $u'$ is chosen to be in the vertex cover, this corresponds to the case where v and w are chosen, if not, u is chosen. It will reduce the problem size to $n-2$ and the parameter size to $k-1$.

Reduction Rule 6

(Complete neighborhood) If G contains a vertex u such that N(u) is a clique (i.e. for every $v,w \in N(u)$ we have $vw \in E$), add N(u) to the vertex cover and remove u and N(u) from the graph.

This is an extension of the Degree Two Adjacent rule above. A complete graph requires all but one of its vertices to cover its edges, and as u is only adjacent to no other vertices, an optimal solution will never require it to cover anything. It will reduce the problem size to $n - |N(u)| - 1$ and the parameter size to $k-1$.

Reduction Rule 7

(Crown Rule Abu-Khzam et al. (2004, 2007)) If G is a graph with a crown (I, H), then there is a vertex cover of G of minimum size that contains all the vertices in H and none of the vertices in I.

A crown decomposition (I, H, B) of a graph G is a partition of V into sets I, B and H such that

• the crown I is a non-empty independent set,

• the head $H=N(I)$, and

• the rest of the graph body $B=V \setminus ( I \cup H )$, and

• there is a matching of size |H| in $G [ H \cup I ]$.

Hence, we can remove all vertices in I and H from the graph G. The problem size is reduced to $n-|I|-|H|$, and parameter to $k-|H|$ after adding H to the vertex cover.

Reduction Rule 8

(Linear-Programming Chen et al. (2001)) Theorem 1 If P, Q and R are defined as below, there is an optimal vertex cover that is a superset of P and that is disjoint from R.

We can formulate the vertex cover problems as an Integer Linear-Programming in the following manner.

For each vertex $u \in V$ we assign a value $X_u \in \{0, 1\}$ such that the following conditions hold:

• Minimize $\sum \nolimits _{u} X_u$

• Satisfy $X_u + X_v \ge 1$ whenever $uv \in E$

As solving integer linear programming is NP-Hard, we use Linear Programming (LP) to approximate the optimal solution for the problem. We can relax constraint $X_u \in \{0, 1\}$ to $X_u \in [0, 1]$, which can further simplified as $X_u \ge 0, \forall {u} \in V$. Hence, the LP formulation for Min Vertex Cover is:

• Minimize $\sum \nolimits _{u} X_u$

• Satisfy $X_u + X_v \ge 1$ whenever $uv \in E$ and $X_u \ge 0$.

To simplify this LP problem, let assume

• N(S) denote the neighborhood of S,

• $P=\{u \in V | X_u > 0.5\}$,

• $Q=\{u \in V | X_u = 0.5\}$ and

• $R=\{u \in V | X_u < 0.5\}$.

Finally, we remove P, R and their adjacent edges from the graph G. The problem size is reduced to $n-|P|-|R|$, and the parameter size becomes $k-|P|$ after adding P to the vertex cover.

Reduction Rule 9

(Struction Chen et al. (2010)) Given a vertex v with neighborhood $N(v) = \{u_{1}, \ldots , u_{p}\}$ and with at most $p-1$ non-edges between its neighbors. For every pair $u_{i}$, $u_{j}$ of non-adjacent vertices in N(v) add a new vertex $u_{i}u_{j}$ with edges to every vertex in $N(u_{i}) \cup N(u_{j}) \setminus \{v\}$. Remove v and N(v) from the graph.

This rule is a generalization of the Degree Two Non-adjacent rule, and reduces the size of the graph to at most $n-1$, and reduces the parameter to $k-1$.

Reduction Rule 10

(General Fold Chen et al. (2010)) Given an independent set I, and its neighborhood N(I) where $|N(I)| = |I| + 1$, with the property that for every $\emptyset \subset S \subseteq I$ $|N(S)| > |S|$, either

• 1. N(I) induces an independent set, we can remove I and N(I), add I to the vertex cover, add a new vertex u and add the edge uv whenever v was a neighbor of some vertex $w \in N(I)$, and reduce k by |I|, or

• 2. N(I) does not induce an independent set, and we may remove I and N(I) from the graph, and add N(I) to the vertex cover and reduce k by N(I).

Current methods and practices of automated heuristics design

In relation to the toolkit of techniques to design metaheuristics, Crainic and Toulouse (2003) in Crainic and Toulouse (2003) presented a survey on parallel metaheuristic developments. It mainly focused on the parallel design and implementation principles of metaheuristics for larger problems to solve in reasonable computing times. It presented the parallel design principles for genetic algorithm, simulated annealing and tabu search. Birattari et al. (2006) reviewed about the analysis of problem of evaluating metaheuristics proposed in theory and used in practice Birattari et al. (2006). The experimental practice in machine learning for evaluating the performances of metaheuristics were mainly reported in this article. Zhang et al. (2017) edited a special issue of “Journal of Optimization” Zhang et al. (2017), focused on various applications of algorithmic design for metaheuristic optimization algorithms. We found following complex metaheuristics (mostly dependent on population-based search techniques) being applied on some classical and real-life problems:

• Genetic Algorithm: The Genetic Algorithm (GA) was used for solving the Minimum Dominating Set of Queens Problem and image processing problem to detect spots or disease on the plant.

• Particle Swarm Optimizers: Hybrid Particle Swarm Optimizers (PSO) was applied for software engineering optimization in MapReduce programming, identifying genetic signature for cancer classification and for solving (with help of memetic algorithm) a complex military problem.

• Memetic Algorithm: Memetic algorithm (MA) is popular in combinatorial optimization problems Berretta et al. (2003); França et al. (1999); Moscato et al. (2010); Naeni et al. (2014); Moscato (2012); Berretta et al. (2012) and Multi-objective variation of memetic algorithms were employed to solve an examination timetabling problems and a complex real-world military problem (in conjunction with hybrid PSO to solve the problem).

• A* Search: Sparse A* Search (SAS) was used for unmanned combat aerial vehicle (UCAV) path planning problem.

Sörensen et al. (2008) in Sörensen et al. (2018) presented a brief history of metaheuristics for five development periods. The authors highlighted the paradigm shift in development of heuristic methods from method-centric to framework-centric period and to further explore the succession into the scientific period. They expected to see more work in the research field of this scientific period to generate structure knowledge to benefit both the researchers and practitioners. Zufferey (2012) in Zufferey (2012) proposed a set of 17 rules to use for metaheuristics design. Among them, eight rules being proposed for designing the metaheuristic. They also proposed rules for designing local search and evolutionary methods for metaheuristics. The author illustrated each of the rules for three types of well-known optimization problems: graph coloring, vehicle routing and job-shop scheduling problems. Their analysis of the literature showed that the complex metaheuristic methods which combined the population search and efficient local search procedures were seemed to be the most promising. Nakib et al. (2017) in Nakib et al. (2017) proposed a complex framework to design metaheuristic using machine learning method evolved on the mutual information metric until the stopping criteria were meet. The maximum likelihood principle was used to implement the framework. The method was tested only on a set of functions in the literature of large scale continuous optimization.

More recent reviews by references Stützle and López-Ibáñez (2018, 2019); Hussain et al. (2019) summarised the latest advances in metaheuristic algorithms. Among them, Stützle and López-Ibáñez (2019) focused on the automatic design and configuration of metaheuristics algorithms and Hussain et al. (2019) presented a comprehensive survey of metaheuristics for 33 years (starting from 1983 to 2006) in Ref. Hussain et al. (2019). In their both publications Stützle and López-Ibáñez (2018, 2019), Stützle and López–Ibáñez mainly focused on the drawbacks of the manual, labor-intensive and intuition-based approach of algorithm design, and the recent advancement on automatic design and configuration of metaheuristic algorithms. However, in Hussain et al. (2019) the authors discussed the trends of metaheuristics by application area, types of metaheuristics and the theoretical and mathematical foundations of metaheuristic design.

All of these works of literature presented the methods and practices of automated heuristics design. We have found mainly two basic approaches of heuristic design and configuration. The first category is self-tuning and self-adapting heuristics driven by search techniques (local search and/or population-based method, e.g., Ref. Zufferey (2012)). The other group learns from a set of training instances and then generalises to unseen instances (often denoted as the Machine Learning-based approach, e.g., in Nakib et al. (2017)). There are some methods which combined both approaches (it is worth noting that Zufferey (2012) found the combined methods to be the most promising) to create complex heuristic.

In contrast, the more practically oriented field of Heuristics, and particularly the modern practice of developing metaheuristics employs a much more experimentally driven approach, including toolkits based around adversarially developed heuristics, machine learning and so forth. Unfortunately, the loop back to Algorithmics (i.e. mathematical procedures that have theorem-proven guarantees) is not closed, and while many well tested and effective heuristics and metaheuristics that work well in practice have been produced Birattari et al. (2006); Crainic and Toulouse (2003); Nakib et al. (2017); Stützle and López-Ibáñez (2019); Zhang et al. (2017); Zufferey (2012), the advancement of the algorithmic understanding of the problem is at best slow.

Current trends in metaheuristics for this problem

We offer here a brief survey of metaheuristics for the vertex cover problem where the results are presented for BHOSLIB and DIMACS benchmarking dataset. However, we note that most of the metaheuristics reported results only for a subset of the instances of the BHOSLIB and DIMACS benchmark datasets.

Guturu and Dantu in Guturu and Dantu (2008) presented an impatient EA with probabilistic tabu search (IEA-PTS) for the Min Vertex Cover (VC) problem. The proposed method works in two stages. First, the problem is mapped onto the maximum clique-finding problem (MCP), which is then solved using an evolutionary strategy. The EA learns not only form previously successful search directions but also from previous failures. The probabilistic tabu-search (PTS approach is used to discourage the search of earlier unfruitful directions. They have used 37 instances form the DIMACS benchmark set for VC problems. Along with the minimum cover size obtained by the IEA-PTS, they also reported the results for two other metaheuristics: a stochastic local search algorithm named Cover Edges Randomly (COVER) Richter et al. (2007) and CycleKernelized Ant Colony System (CKACS) Gilmour and Dras (2006). The COVER Richter et al. (2007) is a Stochastic Local Search (SLS) method for that uses an edge weighting-based heuristic. It starts from a uniformly random initial candidate solution which is iteratively improved by small step thus creating neighboring candidate solutions. The method increases the weights of yet uncovered edges at each step of the iteration. The CKACS is an Ant Colony System (ACS) metaheuristic which continually reinforces the kernelization information as the global pheromone update rule Gilmour and Dras (2006).

A multi-start metaheuristic called Greedy Randomized Adaptive Search Procedure for the variant of the core problem called Connected Vertex Cover (GRASP-CVC) has been proposed in Zhang et al. (2018). We review it here since any feasible solution of this method is also one of the unrestricted version and it can give an idea of the power of this method. Also, some of the heuristics proposed in this paper (in Sect. 4.6) can be useful as part of a GRASP strategy. GRASP-CVC used a greedy function and a restricted candidate list to construct high-quality initial solutions. The initial solution is then iteratively improved and a neighborhood of solutions is explored via a local search mechanism. This method was only tested on a small subset of DIMACS data instances. O. Ugurlu proposed an Isolation Algorithm (IA) Ugurlu (2012) based on isolating the vertex with a minimum degree and followed by the addition of the neighboring vertices of the isolated vertex into in the covering set. They applied the proposed algorithm on the BHOSLIB and DIMACS instances. In Cai et al. (2013), a metaheuristic for VC with two new strategies, called NuMVC, is proposed. First, a strategy for selecting two vertices (one vertex from the current candidate solution for removal, then uniformly at random selects another vertex from uncovered edges for addition) to exchange separately. The exchange of those vertices is performed in two stages: one at the remove stage and another in the add stage. Next, the strategy is to periodically increase and decrease the edge weighting. This algorithm has been applied to both the BHOSLIB and DIMACS instances.

1.1 More complex heuristics on BHOSLIB Dataset

We found several local search based more complex heuristics for the Min Vertex Cover problem which reported results on the BHOSLIB benchmark datasets. The vertex cover size and runtime reported by COVER Richter et al. (2007), CKACS Gilmour and Dras (2006), IEA-PTS Guturu and Dantu (2008), GRASP-CVC Zhang et al. (2018), and the times they took are surveyed here to give some context in comparison with constructive heuristics like IA Ugurlu (2012) and our h3. The results on the BHOSLIB instances are shown in Table 5. Note that, generally speaking, iterative improvement based schemes require orders of magnitude more CPU time.

Table 5 The cover size (C) and runtime (t in seconds) reported for BHOSLIB instaces by the five complex metaheuristics [we used subscripted short name of er for COVER (Richter et al. 2007), ck for CKACS (Gilmour and Dras 2006), ea for IEA-PTS (Guturu and Dantu 2008), as for GRASP-CVC (Zhang et al. 2018) and ia for IA (Ugurlu 2012)]

1.2 Complex metaheuristics on a subset of DIMACS instances

The vertex cover results reported by three metaheuristics (GRASP-CVC, IA, NuMVC) on a subset of DIMACS instances are shown in Table 6.

Table 6 The cover size (C) and runtime (t in seconds) reported in the papers for a subset of DIMACS instaces by the three complex metaheuristics

1.3 Summary of results

From the result on BHOSLIB instances in Table 5, we can see that the COVER approach exhibited comparatively better performances in finding the optimal solution reported by Guturu and Dantu (2008). It has found the exact solution for 18 instances. However, in Richter et al. (2007) the COVER method was being applied on DIMACS instances and it did not perform well on the brock family of graphs where the algorithm failed to escape local minima Richter et al. (2007). In this regard, our heuristic h3 seemed to be an effective constructive method in finding high quality solutions form different types of graphs. They may give researchers new insights about how to improve their metaheuristics (e.g. the heuristic can be used as an initialization approach and to help recombination in evolutionary algorithms).

The compiled results for Min Vertex Cover by different metaheuristics on the subset of DIMACS instances are shown in Table 6. Here we can see that NuMVC ($C_{nu}$) exhibits the best performance. However, the runtimes have not been reported. Only the GRASP-CVC and IA methods reported their runtimes. The runtimes are presented here as a guideline, but they are not comparable, because these algorithms being executed in completely different computers.

1.4 Performance comparison with Isolation Algorithm (IA) proposed in Ugurlu (2012)

O. Ugurlu proposed a simple heuristic, Isolation Algorithm (IA), in Ugurlu (2012), based on adding all the neighbors of the minimum degree vertex into the cover. They presented both the size of vertex cover and the runtime of the heuristic for BHOSLIB and a subset of DIMACS instances. However, we noticed that some of the exact covers reported in the paper do not match known values of the optimal cover. For instance, the reported optimum vertex cover size of 345 for gen400_p0.9_65, is problematic since it is known to be smaller (335). The optimum cover reported for MANN instances are also not matching with NuMVC in Cai et al. (2013). NuMVC reported that 252 and 690 are optimum values for MANN_a27 and MANN_a45, respectively; IA reported them as 375 and 1032. They also reported the best-known cover size of MANN_a81 as 2221; however, IA reported it as 3318. This cover size is more than 1000 vertex larger than the known size. These large differences in the known cover sizes used in the paper presents some problems to directly compare with the published values.

To address this problem, we have implemented the IA algorithm following Ugurlu (2012). However, from their description, it is also unclear how ties are meant to be broken when more than one vertex have the minimum degree. In our implementation of the heuristic IA, we took a vertex uniformly at random from those with the minimum degree. We then executed both our Heuristic 3 and the IA method on BHOSLIB, DIMACS, the small subset of DIMACS Datasets used by Asgeirsson and Stein (2005) and also on seven instances form Network Data.¹⁰ Here we report the summary of the comparison in terms of the number of times H3 wins, loses and tie with IA in Table 7.

Table 7 Number of times the Proposed Heuristic 3 wins, loses and had a tie with Heuristic IA (Ugurlu 2012)

In the work of Cai et al. (2013), the brock graphs are referred as one of the hardest types of instances for vertex cover in DIMACS database. Among the 12 brock instances, IA ties with h3 in 7 cases and h3 wins in the remaining 5 cases. The most difficult instances from DIMACS are C2000.5, MANN_a81, keller6 and MANN_a45 Cai et al. (2013); Grosso et al. (2008); Richter et al. (2007). While we compare the vertex cover sizes found by IA and h3 for these instances, the IA has a tie with h3 for both of the MANN instances, wins on C2000.5, and loses on keller6. These clearly indicate that h3 is competitive with IA.

Finally, to find if h3 is statistically better than IA, we conducted a one-tailed Wilcoxon signed ranked test.¹¹ Among the combined 140 instances from four sources (in Table 7). We found 76 cases of the tie, which yields the number of samples to $N=64$. For those samples with a confidence level $\alpha =0.05$, the one-tailed Wilcoxon Signed-Rank test revealed the W-value=788.5. As the distribution is approximately normal, the z-value is used. The value of z is $-1.6819$ with the p-value = 0.04648. Hence, the result is significant at $p < 0.05$ and “h3 is significantly better or equal than IA”.