AH3: An adaptive hierarchical feature representation model for three-way decision boundary processing

Abstract

Three-way decision theory is an effective method to deal with uncertain data in classification problems. For binary classification, it divides samples into positive, negative and boundary regions (POS, NEG, and BND). The BND region is regarded as a feasible selection of decision-making when the useful information is too limited to make a correct decision, which needs further processing to improve the binary classification. Many existing boundary processing methods are oriented only toward the data and not to the problem. How to obtain effective decision rules from the problem itself to guide boundary division is a challenge. In this article, we propose an adaptive hierarchical feature representation model based on three-way decision theory (named $AH3$) with a problem oriented for boundary processing. We firstly divide all samples into two certain regions (POS and NEG) and an uncertain region (BND) using a three-way decision model based on a minimum covering algorithm. Secondly, we obtain the hierarchical feature representation of POS and NEG regions on the basis of fuzzy quotient space theory, and select the optimal layer with a problem-oriented validating BND region of the training set. Finally, we adaptively decompose the optimal layer between the upper layer and the lower layer to find the adaptive granular space for boundary processing. The experimental results obtained with five University of California, Irivine datasets show that our algorithm effectively increases binary classification accuracy.

Introduction

Classification has always been a key issue in real life, especially in disease detection, spam detection, and many other applications. For traditional binary classification, there are only two choices for a decision: acceptance and rejection. In practice, it is often impossible to accept or reject some samples because of uncertainty or incompleteness of information, and they are often easily misclassified. Thus, how to properly deal with these uncertain samples for binary classification is an important problem. To better describe datasets and process uncertain samples, three-way decision theory was proposed by Yao [1], [2] to extend two-way decision theory by incorporating an additional choice: boundary decision. All samples are divided into three possible decisions: positive decision, negative decision and boundary decision; namely, as a positive region (POS), a negative region (NEG), and a boundary region (BND).

In three-way decision theory, the POS and NEG regions both contain samples without uncertainty or fuzziness, and uncertain samples are further divided into certainty regions (POS or NEG) when they contain enough information. The process is similar to the human decision strategy. Therefore, how to mine information for boundary processing is especially important, and that is the best way to reasonably improve the results of binary classification. Research on three-way decision theory now focuses more on solving practical problems. For example, Chen et al. [3] extended the notions of a characteristic relation and a characteristic set to systems with four types of characteristic relations and characteristic sets for incomplete data processing based on the wide sense of three-way decision theory. Afridi et al. [4] used the three-way clustering approach to deal with clusters having overlapping regions. They proposed different variance-based criteria for determining the thresholds. The thresholds play a crucial and important role in accurate estimation of the overlapping region. In terms of theoretical innovation, Liu et al. [5] proposed a new three-way decision model with intuitionistic fuzzy numbers to solve multiple-attribute decision-making problems. Zhang et al. [6] constructed and investigated quantitative three-way class-specific attribute reducts based on region preservations in three-way decision theory, with the aim to form three-way types of quantitative optimization that match probabilistic rough sets. In addition, the application of three-way decision theory involves various fields, such as disease detection [7], [8], credit card [9], social networks [10], recommendation systems [11], text classification [12], image data analysis [13], inconsistent information [14], [15], and cloud computing [16].

To better solve the binary classification problem, researchers have made contributions for processing the BND region, and many advanced techniques have resulted. Li et al. [17] proposed a method based on the tri-training algorithm to reduce the BND region, and build up three classifiers on the basis of a three-way decision. Li et al. [18] proposed a three-way decision model for dealing with the uncertain boundary region to improve the binary text classification performance based on rough set techniques and a centroid solution. Ma and Yao [19] thought negative rules are as important as acceptance rules for boundary processing, which is based on class-specific attribute reducts. They proposed three types of class-specific attribute reducts in probabilistic rough set models for boundary processing. We have made much effort in dealing with boundary areas. First, we [20] mined definite information from the POS region and the NEG region, and proposed a multiview decision model based on constructive three-way decision theory, which mined the global information to classify boundary samples. Then, we [21] used a cost-sensitive method to deal with the BND region based on the three-way decision model.

However, most of these methods are oriented toward the data themselves and not to toward the problem decision. That is why humans always think about the reasons why problems arise. They find answers from existing knowledge using conventional methods. They may draw different conclusions based on different basic knowledge or criteria. Therefore, on the basis of three-way decision theory, how to mine useful decision rules from the POS region and the NEG region is an important step, and our goal is to use these two kinds of decision rules to properly divide these boundary samples to increase binary classification accuracy.

In this article, we propose an adaptive hierarchical feature representation model based on three-way decision theory for boundary processing, named $AH3$. Our contributions are as follows:

• We propose a BND region processing method (AH3) for problem-oriented decision making. We select the adaptive feature representation for boundary processing using a validated BND region, which properly improves the results of binary classification.

• On the basis of fuzzy quotient space theory (FQST), we construct two kinds of hierarchical feature representation from the POS region and the NEG region, respectively. Then, we adaptively decompose the feature representation with the highest accuracy between the upper layer and the lower layer to finer granularity, and two adaptive granular spaces are selected from the POS region and the NEG region to properly process boundary samples.

• We combine variance with mutual information to form a new method (variance-mutual information [VMI] method) that can better represent the relationship between different features to highlight some representative features and remove some redundant features.

• To demonstrate the effectiveness of our algorithm (AH3), we experiment on five University of California, Irvine (UCI) datasets: the Spambase dataset, the Chess dataset, and three medical datasets: Breast Cancer Wisconsin (Original) (WBC), Breast Cancer Wisconsin (Diagnostic) (WDBC), and Breast Cancer Wisconsin (Prognostic) (WPBC). The results demonstrate that our algorithm has good classification performance, especially in dealing with the three real medical datasets.

The remainder of this work is organized as follows. In Section 2, we introduce our preliminary work. In Section 3, the process of constructing a hierarchical feature representation based on FQST is introduced. In Section 4, we introduce our algorithm of an adaptive hierarchical feature representation model based on three-way decision theory (AH3). The experimental results are analyzed in Section 5. In Section 6, we present our conclusions.