Instance-based weighting filter for superparent one-dependence estimators

Abstract

Bayesian network classifiers remain of great interest in recent years, among which semi-naive Bayesian classifiers which utilize superparent one-dependence estimators (SPODEs) have shown superior predictive power. Linear weighting schemes are effective and efficient ones for linearly combining SPODEs, whereas it is a challenging task for averaged one-dependence estimators (AODE) to find globally optimal and fixed weights for its SPODE members. The joint probability distribution of SPODE may not always fit different test instances to the same extent, thus a flexible rather than rigid weighting scheme would be a feasible solution for the final AODE to approximate the true joint probability distribution. Based on this promise, we propose a novel instance-based weighting filter, which can flexibly assign discriminative weights to each single SPODE for different test instances. Meanwhile, the weight considers not only the mutual dependence between the superparent and class variable, but also the conditional dependence between the superparent and non-superparent attributes. Experimental comparison on 30 publicly available datasets shows that SPODE with instance-based weighting filter outperforms state-of-the-art BNCs with and without weighting methods in terms of zero–one loss, bias and variance with minimal additional computation.

Introduction

Classification is one of the most active research area in both machine learning and data mining communities. Researchers have proposed tremendous classification algorithms [1], [2], [3], [4], [5], among which Bayesian network classifiers (BNCs) receive much attention in recent years, especially after the success of naive Bayes (NB) [6]. However, in practice, the conditional independence assumption of NB rarely holds, and as a result its probability estimates may be suboptimal. A large literature addresses approaches to relax NB’s conditional independence assumption, which can be broadly placed into two categories: semi-naive Bayes approaches and model weighting approaches. The first category is aimed at enhancing the accuracy of NB by introducing a limited number of arcs that represent additional dependence relationships [7], [8]. On the other hand, attribute weighting approaches are usually viewed as a means of increasing the influence of highly predictive attributes and discounting attributes that have little predictive value [9], [10].

Superparent one-dependence estimator (SPODE) [11] is a subcategory of one-dependence estimator which allows all attributes to depend on one superparent except the class variable. Since it is a challenging task for SPODE to find a globally optimal superparent, ensemble methods are often applied to use each single attribute as a superparent to build a SPODE and combine multiple SPODEs for classification. Averaged one-dependence estimator (AODE) [12] is the representative of such SPODE, which has demonstrated excellent classification accuracy with very little extra computational cost. However, the averaging strategy of AODE (i.e., each attribute is treated equally) neglects the different roles of attributes in various learning tasks, which may lead to negative effect on the generalization performance. Existing learning approaches addressing this issue can be broadly placed into four categories:

(1) Attribute weighting [9], [10], [13], [14], [15], [16], [17], [18];

(2) Attribute selection [19], [20], [21], [22];

(3) Model selection [23], [24], [25], [26];

(4) Lazy learning [27], [28], [29], [30].

In this paper, we focus our attention on attribute weighting methods and the linear weighting method can be described as follows,

$$\hat{P}(c,\mathbf{x})=\sum _{i=1}^n{w}_i{P}_i(c,\mathbf{x}).$$

Thus the joint probability of weighted AODE is the summation of weighted joint probabilities of its SPODE members. If the SPODE members are assigned with fixed weights, then the extents to which corresponding joint probability distributions of the SPODE members fit different data points should always remain the same. Fixed weights can be regarded as a globally optimal solution to weighting. However, if a globally optimal solution is very sensitive to perturbations in the data then there may be cases where it is not good to use this solution [31]. Take dataset Phoneme for example, Fig. 1 presents the distributions of the joint probability for SPODEs on every instance. As can be seen, the distributions differ greatly for different SPODEs, and the probability distribution of the same SPODE does not fit different instances to the same extent. Thus, we argue that the fixed weights (globally optimal solutions) may result in biased estimate of joint probability distribution of the weighted AODE and then suboptimal classification performance. Finding locally optimal solution to weighting for different test instances is a feasible solution. Thus, in this paper we propose a novel filter weighting approach, called instance-based weighting, which can self-adaptively assign appropriate weights to the same SPODE while classifying different instances. Meanwhile, the method not only considers the mutual dependence relations between the superparent and class variable, but also takes into account the difference among sub-models. This method can be applied in various SPODE and in this paper we take AODE as an example. We demonstrate experimentally that the instance-based weighting AODE (IWAODE) outperforms other state-of-the-art BNCs with and without weighting methods in terms of zero–one loss, bias and variance without incurring high computation on 30 publicly available datasets [32].

The rest of this paper is organized as follows. Section 2 introduces AODE and provides a brief survey of related weighting approaches. Our proposed novel techniques for instance-based weighting are described in Section 3. Section 4 presents experimental results of IWAODE and its comparison with other state-of-the-art learners. Section 5 draws conclusions.