A novel approach to define the local region of dynamic selection techniques in imbalanced credit scoring problems

https://doi.org/10.1016/j.eswa.2020.113351Get rights and content

Highlights

  • A novel local competence definition for imbalance dynamic selection classification.

  • An evaluation of the classification complexity of credit scoring datasets.

  • A reduction of a dynamic selection technique to a static selection approach.

  • An evaluation of the prediction performance of RMkNN in comparison with regular kNN to imbalanced credit scoring datasets.

Abstract

Lenders, such as banks and credit card companies, use credit scoring models to evaluate the potential risk posed by lending money to customers, and therefore to mitigate losses due to bad credit. The profitability of the banks thus highly depends on the models used to decide on the customer’s loans. State-of-the-art credit scoring models are based on machine learning and statistical methods. One of the major problems of this field is that lenders often deal with imbalanced datasets that usually contain many paid loans but very few not paid ones (called defaults). Recently, dynamic selection methods combined with ensemble methods and preprocessing techniques have been evaluated to improve classification models in imbalanced datasets presenting advantages over the static machine learning methods. In a dynamic selection technique, samples in the neighborhood of each query sample are used to compute the local competence of each base classifier. Then, the technique selects only competent classifiers to predict the query sample. In this paper, we evaluate the suitability of dynamic selection techniques for credit scoring problem, and we present Reduced Minority k-Nearest Neighbors (RMkNN), an approach that enhances state of the art in defining the local region of dynamic selection techniques for imbalanced credit scoring datasets. This proposed technique has a superior prediction performance in imbalanced credit scoring datasets compared to state of the art. Furthermore, RMkNN does not need any preprocessing or sampling method to generate the dynamic selection dataset (called DSEL). Additionally, we observe an equivalence between dynamic selection and static selection classification. We conduct a comprehensive evaluation of the proposed technique against state-of-the-art competitors on six real-world public datasets and one private one. Experiments show that RMkNN improves the classification performance of the evaluated datasets regarding AUC, balanced accuracy, H-measure, G-mean, F-measure, and Recall.

Introduction

Credit offer is a key activity for banks that aim at improving their profitability and competitiveness. Small improvements in the default prediction imply significant profits to the financial institutions Hand and Henley (1997). However, the decision to grant a loan to a customer is complex and risky because it requires an accurate default prediction to protect banks from financial losses, especially during the financial crises. Thomas, Crook, and Edelman (2017) pointed out several aspects affecting the default rate over time, such as the cost of the money (interest rate), the supply and demand for credit, the state of the economy, and the cyclical variations of credit over time. Besides these aspects, data availability, accuracy, and reliability make the default prediction much harder than other domain-specific classification problems. Therefore, new methods and techniques, called credit scoring models, are required to cope with these problems while guaranteeing a low percentage of defaults.

Basel accords regulate credit scoring. Basel Capital Accord II defines the validation and verification of three estimates: the probability of default (PD), loss given default (LGD), and exposure at default (EAD) (Thomas, Oliver, & Hand, 2005). In this paper, as in Feng, Xiao, Zhong, Qiu, and Dong (2018), we look for improvements in the estimate of the probability of default (PD) in existing historical loan data.

Available historical loan data creates an excellent opportunity to take advantage of trending machine learning methods for building accurate credit scoring models. However, in the real world, credit scoring datasets are imbalanced and skewed data is a challenge for machine learning methods since classifiers tend to predict only the majority class.

In the past few decades, researchers have attempted to optimize the predictive performance in imbalanced data. According to Haixiang et al. (2017), the two most used approaches are Resampling, a kind of preprocessing technique that changes the class distribution of the training set, and Ensemble methods, which could combine several base classifiers, resampling, and cost-sensitive approaches. Ensemble methods, also known as Multiple Classifier Systems (MCS), are justified by several theoretical (Dietterich, 2000, Kuncheva, 2002) and empirical (Fernández-Delgado, Cernadas, Barro, Amorim, 2014, Opitz, Maclin, 1999) studies which demonstrate the superiority of ensembles over individual classifier models. They are widely used to solve many real-world problems, including credit scoring (Lessmann, Baesens, Seow, Thomas, 2015, Xiao, Xiao, Wang, 2016), and class imbalance (Galar, Fernandez, Barrenechea, Bustince, & Herrera, 2012).

An ensemble is typically composed of three phases: (1) Pool generation, (2) selection of base classifiers, and (3) integration of predictions Britto Jr, Sabourin, and Oliveira (2014). The main target of the pool generator phase is generating diverse classifiers. The selection phase is responsible for selecting the most competent classifiers to perform the prediction, while the integration phase is responsible for the fusion of the results of all models in the ensemble prediction.

The selection phase can be static or dynamic. The static selection consists of selecting the base models once and use the resulting ensemble to predict all the test samples. In the dynamic selection, the base classifier’s competence in the neighborhood of the query sample is used to select which base models are used to predict each sample. Recently, dynamic selection has received attention from the academic community.

It is worth noticing that, as highlighted by Lessmann et al. (2015), dynamic selection classification techniques might violate regulatory requirements in credit scoring because they use different scorecards for different customers. The motivation for this regulation constraint is to avoid customer discrimination. We believe that this regulatory requirement can change if some work confirms that the use of dynamic classification does not include any customer discrimination. However, the demonstration of the lack of correlation between dynamic classification and customer discrimination is not the aim of this paper. Additionally, the existence of other papers evaluating dynamic classification to credit scoring problem (Ala’raj, Abbod, 2016a, Ala’raj, Abbod, 2016b, Feng, Xiao, Zhong, Qiu, Dong, 2018) encourages us to explore this topic.

Analyzing recent works about credit scoring problem, we find several papers that evaluate the prediction performance of classification approaches for credit scoring datasets, such as (Abellán, Castellano, 2017, Ala’raj, Abbod, 2016a, Ala’raj, Abbod, 2016b, Feng, Xiao, Zhong, Qiu, Dong, 2018, García, Marqués, Sánchez, 2019, He, Zhang, Zhang, 2018, Lessmann, Baesens, Seow, Thomas, 2015, Sun, Lang, Fujita, Li, 2018, Xia, Liu, Da, Xie, 2018, Xia, Liu, Li, Liu, 2017, Xiao, Xiao, Wang, 2016). However, to the best of our knowledge, a combination of preprocessing approaches, dynamic selection techniques, and pool generators ensembles is presented only in Melo Jr, Nardini, Renso, and Macedo (2019b). Nonetheless, this previous work only compares the combination of techniques.

Beyond this gap in credit scoring papers, we do not find scientific papers evaluating the suitability of dynamic selection techniques to credit scoring problem. However, Britto Jr et al. (2014) concluded that dynamic selection is appropriate for complex datasets. This motivates us to evaluate the complexity of credit scoring datasets in comparison to datasets of other domains. If credit data is more complex than the average, this can suggest that dynamic selection may be appropriate for this domain.

This work is partially motivated by the outstanding results achieved recently by the dynamic selection techniques (Britto Jr, Sabourin, Oliveira, 2014, Roy, Cruz, Sabourin, Cavalcanti, 2018). Paper Roy et al. (2018) proposes the use of a bagging pool generator combined with oversampling techniques to reduce the effects of the skewed data. Furthermore, although the literature recommends the use of different data in the dynamic selection dataset (DSEL) and training data, they also use oversampling techniques over the training data to generate the DSEL. They decided to use this approach to avoid the lack of minority samples in the training data and the DSEL. The authors rely on the diversity ability of oversampling techniques to avoid bias results.

However, in this paper, instead of using oversampling techniques to guarantee the diversity between the training dataset and DSEL and to overcome the skewed data, we adopt different strategies for each issue. Next, we explain the problems of the previous approach and our strategies.

The main problem of use oversampling techniques to balance the DSEL is the inclusion of noise in the dataset. Oversampling techniques have been shown to be useful for building imbalanced prediction models in the last fifteen years (Fernández, Garcia, Herrera, & Chawla, 2018). However, the DSEL function needs to determine the competence of the base models, and this means that a noise sample in the DSEL can produce miscalculated competence of the base classifiers. That is the reason why we decided to evaluate new approaches.

To address the diversity between training and DSEL, we use bootstrapping: the use of random sampling with replacement. We evaluate this method since it always uses a subset of available samples to train each base classifier. In this way, each classifier does not know the entire training data. We believe that this characteristic is sufficient to guarantee diversity when using the same dataset to train the base models and as the DSEL.

To address the skewed of the data in the DSEL, we develop a modification in the k-NN algorithm, named Reduced Minority k-NN (RMkNN), used by the dynamic selection techniques to define the local region of a query sample. Dynamic selection techniques use k-NN to select the samples in the DSEL that define the competence level of each base classifier. However, in an imbalanced dataset, the k-NN algorithm selects mainly samples of the majority class, producing a poor base classifiers competency evaluation.

To evaluate the performance of RMkNN, we extend Melo Jr et al. (2019b) comparison, including other combinations of pool generators and preprocessing techniques and testing them on seven datasets. We evaluate several combinations of dynamic selection techniques, sampling approaches, and pool generators to assess the effectiveness of our proposal. More specifically, we aim to answer the following research questions related to the credit scoring problem:

  • RQ1) Are dynamic selection techniques appropriate for imbalanced credit scoring problems?

  • RQ2) Does the RMkNN improve the prediction performance of kNN?

  • RQ3) Does the use of the RMkNN technique - that defines a novel competence region of dynamic selection techniques - improve the classification performance of imbalanced credit scoring datasets?

To answer the questions above, we present and discuss novel contributions that include:

  • a novel local competence definition for imbalance dynamic selection classification.

  • an evaluation of the classification complexity of credit scoring datasets in comparison to the classification complexity of datasets from other domains.

  • an evaluation of the prediction performance of RMkNN in comparison with the regular kNN.

  • a static selection representation of dynamic selection techniques.

Paper Melo Jr et al. (2019b) evaluated the combination of pool generators, re-sampling approaches, and dynamic selection techniques. That paper shows a simple comparison of the influence of different re-sampling and dynamic selection techniques on the performance of pool generators in imbalanced credit scoring datasets, in contrast to the present paper where we propose three new research questions and introduce a new method to identify a new local region of the query sample to define the competence of each base classifier in a dynamic selection approach.

The organization of this paper is as follows. Section 2 reviews the literature about credit scoring, imbalance learning approaches, and dynamic selection techniques. Section 3 includes a brief description of the classification approaches used in this paper. Section 4 presents the first contribution of this paper that is an evaluation of the suitability of dynamic selection for credit scoring problems. This section also presents the main contribution of this paper that is the Reduced Minority k-NN (RMkNN) technique. Section 5 presents the experimental setup used. Section 6 shows the experimental results. Finally, the last section is dedicated to the conclusion and future work. The online appendix1 provides details of the results.

Section snippets

Background and related work

This study involves four main elements: credit scoring, imbalanced learning, pool generators, and dynamic selection classification. Next, we present the credit scoring related works and the background of pool generators, imbalanced learning, and dynamic selection classification.

Overview of classification techniques

This study aims to evaluate the performance of a novel dynamic selection approach for imbalanced credit scoring datasets over a wide range of classification techniques. For the propose of this study, two sampling approaches, six credit scoring benchmarks, and eight imbalanced ensembles have been selected based on previous credit scoring papers (Brown, Mues, 2012, Melo Jr, Macedo, Nardini, Renso, 2019a).

Reduced Minority kNN for dynamic selection techniques

The main purpose of the dynamic selection dataset (DSEL) in a dynamic selection technique is to introduce the measurement of the competency level of each classifier in each part of the feature space. These parts are called the local regions. The neighbors of a query sample define a local region, and k-Nearest Neighbors(kNN) is used to find them. These samples are used to evaluate the competence of each base classifier of the ensemble. Finally, the prediction procedure uses only the most

Experimental setup

This paper evaluates a novel approach to define the local region used to compute the competence of base classifiers in imbalanced datasets. We now present the experimental setup used to evaluate our proposal.

Experimental results

We now present the results by answering each research question. First, we analyze if the dynamic selection techniques are appropriate to credit scoring datasets. Then, we analyze the differences between performance measures. Finally, we compare the proposed approach with dynamic ensemble approaches that use DSEL generated by preprocessing techniques and the static ensembles.

As in previous works (Abellán, Castellano, 2017, Lessmann, Baesens, Seow, Thomas, 2015), we use the average rank of the

Conclusions and future work

In this work, we present a study of the credit scoring problem. We assess the combination of Dynamic Selection (DS) methods, data preprocessing, and pool generation ensembles to deal with the imbalanced nature of the credit scoring data sets using a novel approach to define the local regions of a dynamic selection technique.

We propose RMkNN to perform a balanced selection of DSEL samples in a dynamic selection technique. To assess the performance of our technique, we compare our proposal with

CRediT authorship contribution statement

Leopoldo Melo Junior: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing - original draft, Writing - review & editing, Visualization, Project administration. Franco Maria Nardini: Conceptualization, Methodology, Validation, Writing - review & editing. Chiara Renso: Conceptualization, Methodology, Validation, Writing - review & editing. Roberto Trani: Conceptualization, Methodology, Investigation. Jose Antonio Macedo: Conceptualization,

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgement

This paper is partially supported by the BIGDATAGRAPES (EU H2020 RIA, grant agreement no. 780751), the OK-INSAID (MIUR-PON 2018, grant agreement no. ARS01_00917), the FUNCAP SPU (grant agreement no. 8789771/2017), the CNPQ (grant agreement no. 309350/2018-2), and the MASTER (EU H2020 Marie Sklowdoska-Curie, grant agreement no. 777695) projects.

References (52)

  • A.H. Ko et al.

    From dynamic classifier selection to dynamic ensemble selection

    Pattern Recognition

    (2008)
  • S. Lessmann et al.

    Benchmarking state-of-the-art classification algorithms for credit scoring: An update of research

    European Journal of Operational Research

    (2015)
  • A. Roy et al.

    A study on combining dynamic selection and data preprocessing for imbalance learning

    Neurocomputing

    (2018)
  • C. Serrano-Cinca et al.

    The use of profit scoring as an alternative to credit scoring systems in peer-to-peer (p2p) lending

    Decision Support Systems

    (2016)
  • M.R. Sousa et al.

    A new dynamic modeling framework for credit risk assessment

    Expert Systems with Applications

    (2016)
  • J. Sun et al.

    Imbalanced enterprise credit evaluation with dte-sbd: Decision tree ensemble based on smote and bagging with differentiated sampling rates

    Information Sciences

    (2018)
  • D. West

    Neural network credit scoring models

    Computers & Operations Research

    (2000)
  • Y. Xia et al.

    A novel heterogeneous ensemble credit scoring model based on bstacking approach

    Expert Systems with Applications

    (2018)
  • H. Xiao et al.

    Ensemble classification based on supervised clustering for credit scoring

    Applied Soft Computing

    (2016)
  • R. Barandela et al.

    New applications of ensembles of classifiers

    Pattern Analysis & Applications

    (2003)
  • L. Breiman

    Bagging predictors

    Machine Learning

    (1996)
  • A.S. Britto Jr et al.

    Dynamic selection of classifiersa comprehensive review

    Pattern Recognition

    (2014)
  • N.V. Chawla et al.

    Smote: synthetic minority over-sampling technique

    Journal of Artificial Intelligence Research

    (2002)
  • C. Chen et al.

    Using random forest to learn imbalanced data

    University of California, Berkeley

    (2004)
  • T. Chen et al.

    Xgboost: A scalable tree boosting system

    Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining

    (2016)
  • R.M.O. Cruz et al.

    DESlib: A dynamic ensemble selection library in Python

    arXiv preprint arXiv:1802.04967

    (2018)
  • Cited by (50)

    View all citing articles on Scopus
    View full text