On the class overlap problem in imbalanced data classification

Abstract

Class imbalance is an active research area in the machine learning community. However, existing and recent literature showed that class overlap had a higher negative impact on the performance of learning algorithms. This paper provides detailed critical discussion and objective evaluation of class overlap in the context of imbalanced data and its impact on classification accuracy. First, we present a thorough experimental comparison of class overlap and class imbalance. Unlike previous work, our experiment was carried out on the full scale of class overlap and an extreme range of class imbalance degrees. Second, we provide an in-depth critical technical review of existing approaches to handle imbalanced datasets. Existing solutions from selective literature are critically reviewed and categorised as class distribution-based and class overlap-based methods. Emerging techniques and the latest development in this area are also discussed in detail. Experimental results in this paper are consistent with existing literature and show clearly that the performance of the learning algorithm deteriorates across varying degrees of class overlap whereas class imbalance does not always have an effect. The review emphasises the need for further research towards handling class overlap in imbalanced datasets to effectively improve learning algorithms’ performance.

Introduction

Learning from datasets with skewed class distributions remains a challenge in machine learning. such datasets are realised as imbalanced datasets and widely seen in many applications, for example, anomaly detection [1], medical prediction [2], [3], [4], object recognition [5], [6] and business management [7]. In these domains, the minority class is usually the class of interest and has a higher misclassification cost than the majority class. Standard learning algorithms generally build classification models based upon the maximum accuracy, which often leads to biased classification towards the majority class and misclassification of minority class instances [8], [9]. However, such failure in classification of imbalanced datasets is not always caused by class imbalance solely. In fact, a linearly separable dataset can be perfectly classified by a typical classification algorithm no matter how skewed the class distribution is [10]. On the contrary, when class overlap is present, even a balanced dataset can be difficult for a learning task.

When dealing with classification of imbalanced data, rebalancing class distribution is among the most common approaches that researchers consider. Many traditional and recent resampling methods only aim at getting a more balanced version of the training data and do not factor in the problem of class overlap [11], [12], [13]. Some methods deal with instances in the overlapping region, especially those near the borderline areas; however, their resampling rates are controlled by the degree of class imbalance [14], [15]. Thus, in some scenarios, results can be highly influenced by class imbalance rather than class overlap. For instance, when a dataset suffers from high class overlap but its classes are slightly imbalanced, insufficient resampling may result in class overlap not being properly addressed. On the other hand, with low class overlap and high class imbalance, excessive resampling may occur.

The impacts of class imbalance, class overlap and other characteristics such as small disjunct and dataset size have been investigated [16], [17], [18], [19]. Class overlap frequently shows the highest negative influence among potential factors including class imbalance [9], [18]. This raises some important questions in handling classification of imbalanced datasets: (1) Are the solutions that mainly aim to rebalance the class distribution sufficiently effective? (2) Should the problem of class overlap be the main concern in developing new algorithms?

Although several reviews on the problem of imbalanced data in classification exist [9], [20], [21], the problem of class overlap in imbalanced data was not emphasised as the main issue and the discussions often lacked support of sufficient experimental evidence. Das et al. [9] proposed that the two key challenges for standard learning algorithms are class imbalance and class overlap. Possible nature of learning outcomes in different scenarios of class imbalance and class overlap based on the dataset size was suggested; however, no experimental evidence was given. The authors also investigated other data irregularities such as small disjunct and missing features; thus, the discussion on the class overlap problem was limited. In [20], merely a brief description of other studies on the effect of class overlap in relation to class imbalance was given. The authors paid particular attention to the discussion of different techniques used in existing methods for handling imbalanced classification. Stefanowski [21] motivated the research community to develop new algorithms for imbalanced data that realise data factors, which included overlapping between classes. The author presented the analyses on characteristics of the minority class, which was divided into sub-regions of safe, borderline, rare and outlier samples. This was studied along with the behaviours of different learning algorithms; however, this cannot yet be mathematically verified on real-world datasets. Like in many other reviews [22], [23], Kaur et al. [24] conducted a comparative analysis of methods, which was mainly organised as data preprocessing and algorithmic approaches, and the problem of class overlap was barely discussed. Some other reviews focused on the issue of imbalanced data classification in specific contexts such as big data [25], [26], multi-class problem [25], [27] and neural networks [28], [29]. These clearly show that there is still a gap in the study of class overlap in the context of class imbalance.

In this paper, the importance of handling class overlap in imbalanced data classification is investigated. This was carried out through an extensive experiment and a critical review of solutions to imbalanced learning. The experiment provides an objective measurement of the impact of class overlap versus the impact of class imbalance. Unlike in previous studies [16], [17], [18], [19], which were based on limited ranges of class imbalance and class overlap degrees, we carried out a full-scale experiment using over 1,000 synthetic datasets. The in-depth review of existing solutions to classification of imbalanced datasets is presented in an alternative perspective rather than data and algorithm levels, which was commonly arranged in other review papers [8], [9], [20], [30], [31]. We considered the main objective of the solutions and categorised them into class distribution-based and class overlap-based approaches for better comparing and contrasting the two approaches. Class distribution-based methods mainly concern and aim to suppress the problem of imbalanced class distribution. Class overlap-based methods focus on improving the visibility of instances, especially positive instances, in the overlapping region. In addition, recent and emerging methods that do not particularly deal with the class imbalance or class overlap problems are also discussed. These include, for example, the use of one of the latest techniques in machine learning, Generative Adversarial Networks (GANs) [32], [33].

The main contributions of this review are listed below.

• A technical discussion with advantages and disadvantages of evaluation metrics including how some of them can be misleading in certain imbalanced contexts

• An extensive experiment illustrating the scales of impact of class overlap and class imbalance on imbalanced dataset classification

• A critical discussion of methods and literature selected from leading peer-reviewed publications in the perspective of class overlap-based and class distribution-based approaches, as well as recent emerging technologies

• An overview of benchmarking methods in the literature showing commonly-used ones that can be considered as good standards, but at the same time suggesting a need for comparing against recent and state-of-the-art methods for more convincing and reliable evaluation

The remainder of this paper is organised as follows. In Section 2, we give the definitions of class imbalance and class overlap. Section 3 contains an in-depth discussion of evaluation metrics used in imbalanced learning. Section 4 provides the experimental results and discussion on the effects of class imbalance and class overlap on the learner’s performance in an extensive range of scenarios. In Section 5, we critically review existing approaches for handling classification of imbalanced datasets. Finally, the conclusion is delivered in Section 6.