Non-numerical nearest neighbor classifiers with value-object hierarchical embedding

Highlights

• A value-object hierarchical distance measure for categorical data.

• Two nearest neighbor classifiers applied for categorical data classification.

• Better results of the new classifiers compared with state-of-the-art classifiers.

• Detailed analysis of the impact of factors on the new classifiers.

Abstract

Non-numerical classification plays an essential role in many real-world applications such as DNA analysis, recommendation systems and expert systems. The nearest neighbor classifier is one of the most popular and flexible models for performing classification tasks in these applications. However, due to the complexity of non-numerical data, existing nearest neighbor classifiers that use the overlap measure and its variants cannot capture the inherent ordered relationship and statistic information of non-numerical data. This phenomenon leads to the classification limitation of nearest neighbor classifiers in non-numerical data environments. To overcome this challenge, we propose a novel object distance metric, i.e., value-object hierarchical metric (VOHM), which is able to capture inherent ordered relationships within non-numerical data. Then, we construct two nearest neighbor classifiers, i.e., the value-object hierarchical embedded nearest neighbor classifier (VO-kNN) and the two-stage value-object hierarchical embedded nearest neighbor classifier (TSVO-kNN), which take advantages of both VOHM and non-numerical feature selection. Experiments show that both VO-kNN and TSVO-kNN could mine more knowledge from data and achieve better performance than state-of-the-art classifiers in non-numerical data environments.

Introduction

The classification problem (Murphy, 2012) is a fundamental issue in the areas of machine learning, data mining, artificial intelligence, and expert systems etc. It plays an essential role in many applications, including sentiment analysis, spam filtering, image analysis, text analysis, and DNA sequence analysis, etc. There exist a lot of methods to solve classification tasks such as logistic regression (LR) (Murphy, 2012, Walker, Duncan, 1967), random forest(RF) (Breiman, 2001, Ho, 1995), support vector machines(SVMs) (Catanzaro, Sundaram, Keutzer, 2008, Cortes, Vapnik, 1995), artificial neural networks(ANN) and deep learning(DL) (Bengio, Courville, Vincent, 2013, Lecun, Bengio, Hinton, 2015) etc., which have achieved great success in numerical environments. However, it is a big challenge to apply these models to solve classification tasks in non-numerical environments due to the lack of useful non-numerical distance metrics for evaluating the relations between objects.

Non-numerical (or categorical) data is a widely used data type in expert systems. For example, Table 1 is a staff table instance with non-numerical values. It is convenient for people to acquire knowledge from non-numerical values. However, most existing machine learning algorithms are difficult to deal with the same problem as humans do, because, in the perspective of machine learning algorithms, non-numerical data does not contain the same semantics or context that humans can easily capture and understand. Hence, narrowing the gap between algorithms and humans has become one of the critical factors to improve the classification performance of the algorithms in non-numerical data environments. To solve the non-numerical classification problem, the most commonly used strategy is to convert non-numerical data into numerical data and process it with machine learning algorithms (Buttrey, 1998) such as nearest neighbor classifiers (Chen, Guo, 2015, Hu, Yu, Xie, 2008, Liu, Cao, Yu, 2014). The k-nearest neighbor classifier (kNN) is one of the most frequently used nonparametric classification models in expert systems (Mller, Salminen, Nieminen, Kontunen, Karjalainen, Isokoski, Rantala, Savia, Vliaho, Kallio, Lekkala, Surakka, 2019, Rodger, 2014), because the model has no training phase and easy implementation. Therefore, the problem of how to convert non-numerical data into numerical values becomes the bottleneck of the kNN classifiers for performing non-numerical classification tasks.

The overlap metric (or simply match coefficients) (Boriah, Chandola, & Kumar, 2008) is a commonly used discrepancy measure which is able to convert data from non-numerical form into numerical form. However, due to the neglect of the different contributions of attributes in non-numerical classification problems, the overlap metric can not meet the requirement of real-world applications. Although the attribute weighting (Chen, Guo, 2015, Chen, Ye, Guo, Zhu, 2016, Morlini, Zani, 2012) based approach satisfies these requirements, it still fails to capture latent ordered relationships that exist in non-numerical values. For example, in Table 1, according to common sense, the distance of values between ‘Bachelor’ and ‘Doctor’ is obviously greater than that between ‘Bachelor’ and ‘Master’. However, capturing this latent ordered information is beyond the scope of the overlap measure and attribute weighting based measures. Therefore, when the classification algorithm processes non-numerical data, how to mine latent ordered information from non-numerical values becomes one of the basic problems to be solved. Furthermore, there also exists another complicated dependency relationship between non-numerical values from different attributes. For example, the association rule ‘Beer’ ⇒ ‘Diapers’ found in the sales data would indicate that if a customer buys beer, it is likely also to buy diapers with a certain probability (Agrawal, Srikant, 1994, Han, Pei, Yin, 2000, Zaki, 2000). In other words, It means that there exist dependency relationships between non-numerical values from different attributes. Hence, another question is arose, namely, how to represent and capture the second relationship in non-numerical data environments. In summary, these challenges put new requirements to update the distance measure for non-numerical data. Data science (Cao, 2017a, Cao, 2017b) shows that there exist complicated relationships in data, and a lot of knowledge required intelligence algorithm to discover.

To overcome these problems, in this paper, we construct a novel distance metric, called value-object hierarchical metric (VOHM), to update the strategy of distance measure for non-numerical data. VOHM could learn the latent ordered relationship from the value-object hierarchy structure. At the value level, the VOHM handles all values in a probability perspective, which could capture the latent ordered relationship of values relative to the class label distribution. And, at the object level, VOHM treat the distance of each object pair as the total sum of the discrepancies from the value level. Then, we proposed a nearest neighbor classifier that takes advantages of both VOHM and attribute reduction. Firstly, to avoid the curse of dimensionality and reduce calculation, we employ the rough set theory based attribute reduction strategy to select the required attributes. Then, we equip up the nearest neighbor classifier with VOHM to perform the classification task for the selected attribute filtered data set. More specifically, we developed two nearest neighbor classifiers, i.e., the value-object hierarchical embedded nearest neighbor classifier (VO-kNN) and the two-stage value-object hierarchical embedded nearest neighbor classifier (TSVO-kNN).

The main contributes of this paper are summarized as follows:

• A new distance measure for non-numerical data, i.e., VOHM, is proposed. VOHM enables us to capture the latent ordered relationship, which gives more knowledge than the overlap measure and the weighted overlap measure.

• We propose a value-object hierarchical embedded nearest neighbor classifier that takes advantage of VOHM.

• We propose a two-stage value-object hierarchical embedded nearest neighbor classifier to perform the classification task on the reduced data set to avoid the curse of dimensionality.

The rest of the paper is organized as follows. In Section 2, we briefly do an overview of existing classifiers for non-numerical data. In Section 3, we formulate the research problem and introduce the preliminary notions. Section 4 defines the VO-kNN classifier. In Section 5, we design a feature section algorithm based on rough set theory. In Section 6, we conduct experiments to show the advantage of our model and algorithm. Finally, we conclude this work in Section 7.