Several alternative term weighting methods for text representation and classification

Abstract

Text representation is one kind of hot topics which support text classification (TC) tasks. It has a substantial impact on the performance of TC. Although the most famous TF–IDF is specially designed for information retrieval rather than TC tasks, it is highly useful in the field of TC as a term weighting method to represent text contents. Inspired by the IDF part of TF–IDF which is defined as the logarithmic transformation, we proposed several alternative methods in this study to generate unsupervised term weighting schemes that can offset the drawback confronting TF–IDF.​ Moreover, owing to TC tasks are different from information retrieval, representing test texts as a vector in an appropriate way is also essential for TC tasks, especially for supervised term weighting approaches (e.g., TF–RF), mainly due to these methods need to use category information when weighting the terms. But most of current schemes do not clearly explain how to represent test texts with their schemes. To explore this problem and seek a reasonable solution to these schemes, we analyzed a classic unsupervised term weighting method and three typical supervised term weighting methods in depth to illustrate how to represent test texts. To investigate the effectiveness of our work, three sets of experiments are designed to compare their performance. Comparisons show that our proposed methods can indeed enhance the performance of TC, and sometimes even outperform existing supervised term weighting methods.

Introduction

Automatic text classification (TC) technology can efficiently organize and categorize text that increasing dramatically [1], thus it eliminates a large amount of human effort [2] and attracted a wide attention in recent years [3], [4]. The goal of the TC task is to categorize unlabeled texts into a predefined class based on their topics [5], and hence based on a set of prelabeled texts, an automatic text classifier can be established in the learning process [6], [7]. Before applying classifiers, every term in the text first need to be assigned numerical values (weights) in an appropriate term weighting scheme that is called text representation [8], [9]. The vector space model (VSM) is the most popular way to represent texts [10], [11], it usually treats a text as a set of terms namely bag of words (BoW) model [12], [13]. VSM constructs the text collections as a document-term matrix, in which the term weight represents importance of a certain term ${\mathit{t}}_j$ in a certain document ${\mathit{d}}_k$ [14], [15], and each row denotes one of the document vectors, whereas each column corresponds to one of the distinct terms (i.e., selected features).

Term weighting is critical to TC task that has a direct and significant effect on the text classification performance [16]. At present, term weighting approaches are generally grouped into unsupervised and supervised according to whether they embrace the class information of training texts [17], [18]. The unsupervised term weighting (UTW) methods neglects the diversity of class information, whereas the supervised term weighting (STW) methods exploit the category information when calculating the weight. For UTW schemes, such as term frequency (TF) and TF–IDF (term frequency–inverse document frequency) are commonly used. Among them, TF is one of the simplest weighting methods, but it is a local weighting approach due to it only considers how many times a term occurs in the text. To conquer this drawback, an inverse document frequency (IDF) has been designed to generate the TF–IDF scheme, it is concerned with how many texts a term has appeared in. Note that, TF–IDF has been primarily designed for information retrieval (IR) rather than TC tasks [10], [19].

Different from IR task, TC task aims to discriminate between different classes, not texts [20], and thus it should take category factors into account when computing the weight of terms. For that reason, it can be said that TC is a supervised learning task [9], [17], [21], [22]. Recently, most STW methods are originated from feature selection schemes, these methods adopt the category information in several ways, and can be summarized as follows. First of all, TF-CHI2, TF-IG and TF-GR have been proposed based on feature selection approaches (i.e., Chi-square statistic (CHI2), information gain (IG) and gain ratio (GR)) [18]. Since then, various STW methods similar to the above schemes have also been presented, for example, odds ratio (OR) weighting factor in TF-OR [14], [23], [24], mutual information (MI) weighting factor in TF-MI [14], [23], probability-based (PB) weighting factor in TF-PB [24], and correlation coefficient weighting factor in TF-CC [24].

Apart from these schemes, a variety of STW schemes have been built and proposed, which are derived from TF–IDF. Initially, inspired by the IDF in TF–IDF scheme, the inverse class frequency (ICF) has been introduced, it indicates that a key term of the specific class usually appears in only a few categories [25]. However, due to the number of categories is generally quite small, a certain term may occasionally exist in multiple categories or sometimes even in all categories [25], [26]. As a result, ICF fails to promote the degree of importance of a term under certain circumstances. To enhance a term’s distinguishing power, the ICF has been incorporated to generate the TF–IDF–ICF scheme [14], [25]. Meanwhile, in [14], the authors pointed out that TF–IDF–ICF scheme emphasizes on rare terms. To alleviate this problem, they redesigned the ICF and proposed a novel scheme called TF–IDF–ICSDF. Besides these, Lan et al. [17] proved that the distinguishing abilities of a term depends only on the relevant texts that contain this term, and hence they presented a novel STW scheme by replacing IDF in TF–IDF with the RF (relevance frequency). More importantly, the results in many literatures demonstrated that TF–RF achieves better performance than most STW and UTW approaches [9], [17], [23]. Moreover, in [22], the authors claimed that TF–IDF was not necessarily suitable for TC task, hence two novel term weighting method called TF-IGM and RTF-IGM were presented. Further, to deal with the drawback of TF-IGM, two improved weighting methods based on IGM have also been developed [27].

We must note, however, that not all STW schemes are always superior to UTW schemes [17], [20], [26], [28]. Moreover, as emphasized in [22], most STW schemes show its own superiority in some specific TC tasks, but in fact they cannot consistently yield the best classification performance except that they require more storage space and running time [14]. Because of these, the motivation of this paper is to focus on UTW methods. We have derived inspiration from the IDF part of TF–IDF which is defined as the logarithmic transformation, and many works are seeking to build a theoretical basis for it [29]. Hence, we naturally doubt whether there are other nonlinear transformation forms like logarithmic transformation that can achieve better classification performance than existing ones? This is the first question we wish to address in this research. Besides that, due to the test texts do not have any prior category label information [30], [31], thus how to properly represent test texts is a difficult but significant task [30], especially for the STW schemes due to these methods need to use category information in the weighting process, these include TF–RF [17], TF–IDF–ICSDF [14], TF-IGM [22], RTF-IGM${}_{imp}$ [27] and so on. Nevertheless, it is not clearly explain how to represent test texts in most of existing schemes [31] including TF–IDF. Naturally, we raise the second question: How to represent test texts for TC task? Besides these two questions, the third question will be raised in Section 3. And the answer will be presented at the end of this article. Our contributions in this work can be summarized in the following: firstly, several nonlinear transformation methods are introduced and compared with the famous TF–IDF which adopts the logarithmic ratio. Secondly, we clearly explained how to represent test texts in the weighting process through a classic UTW method and three typical STW methods with different characteristics. Thirdly, a nonlinear transformation method is designed and used for TF, which performs better than the square root function-based TF. Finally, we conduct an extensive experimental comparison of our schemes with existing UTW and STW methods. Experimental results demonstrate the effectiveness and superiority of our proposed schemes.

The rest of this manuscript is outlined as follows. Section 2 takes several existing term weighting methods as examples to clearly show how to represent test texts. In Section 3, we propose four UTW methods for TC. The experimental settings are explained in Section 4. In Section 5, our findings are presented and discussed. We conclude our work in Section 6.