An extension of the type-1 and singleton fuzzy logic system trained by scaled conjugate gradient methods for multiclass classification problems
Introduction
Multiclass classification problems (MCPs) represent a principal branch in the machine learning field and are formulated as the discrimination of patterns in more than two classes (discrimination between only two classes is referred to as binary classification). Such problems exist in numerous research fields, including the biomedical [1], surveillance and security [2], computer vision [3], [4], [5], aeronautical [6], and industrial [7] fields.
In general, models for solving MCPs can be divided into two categories: the binary classifier (BCs) (i.e., classifiers with a single output) and multiclass classifier (i.e., classifiers with multiple outputs). A BC can solve MCPs using various strategies. The simplest and most commonly applied strategies are called one-versus-one (OvO) [8] and one-versus-all (OvA) [9]. The OvO strategy forms pairs of classes by decomposing the original set of classes into several binary subsets of classes, requiring the use of classifiers, where is the number of classes in the problem. The OvO strategy utilizes a decision stage that predicts classes using voting scheme to incorporate the results of all classifiers, where the class with the highest number of votes is chosen. In the OvA strategy, a classification problem is decomposed into binary problems, where each problem is formulated to distinguish one class from all other classes. Several studies have shown that the OvO strategy is superior to the OvA strategy [10], [11]. However, this advantage is attenuated to some extent because increased computational costs are associated with an increasing number of classes.
Regarding MCPs solved by fuzzy logic systems (FLSs) belonging to the first group of models, it is necessary to implement decomposition strategies because FLSs only generates single outputs. When using such strategies, researchers have largely attempted to improve the performance of FLSs by focusing on improved training methods. The authors of [12] proposed an adaptation of the inference method in a fuzzy association rule-based classification model with the goal of producing more accurate aggregations to improve the classification performance using the OvO and OvA strategies. The authors of [13] implemented an FLS-based classification technique with a binary decomposition strategy (i.e., OvO and OvA) to handle MCPs. They attempted to identify optimal methods for the decision stage of classification. Additionally, the authors of [14] improved classification performance for MCPs by using the conjugate gradient (CG) training method, which incorporates the second-order information to train an FLS. However, these methods require large numbers of fuzzy rules to implement binary decomposition strategies. To avoid this issue, it is necessary to design an FLS with multiple outputs.
In an attempt to overcome the limitation of FLSs for handling MCPs (i.e., the fuzzy classifier has one output and needs to implement a binary decomposition strategy), we propose an extension of type-1 and singleton FLSs (T1-FLSs) called T1-FLS with multiple outputs (T1-FLSMO). The most important aspect of T1-FLSMO is that it can handle MCPs using only a single classifier. Consequently, it can significantly reduce the computational complexity of the FLS-based classifiers compared to the use of OvO or OvA strategies. Furthermore, T1-FLSMO was developed based on the FLS presented in [15], where we introduced the scaled CG (SCG) and SCG using the differential operator (SCGR) for training the FLS, which use second-order information without computing a Hessian matrix. Therefore, the equations for computing the exact value of the Hessian matrix by directional vector using the differential operator for T1-FLSMO are designed to facilitate use of the SCGR training method.
The main contributions of this work can be summarized as follows:
- •
An extension of T1-FLS to facilitate multiple outputs is proposed, enabling it to handle multiclass classification problems and avoid binary decomposition strategies, reducing computational complexity;
- •
The deduction of equations to calculate the gradient for T1-FLSMO is presented, facilitating the use of first-order information training methods available in the literature [14], [15];
- •
The deduction of the equations to compute the exact value of using the differential operator for T1-FLSMO is presented, facilitating to use second-order information training methods;
- •
A comparison of T1-FLS and T1-FLSMO using the steepest descent (SD) and SCG training methods is presented. Additionally, the SCG and SCGR training methods are implemented for T1-FLSMO, reducing the dependence of heuristic parameters choices present in training methods [15], thereby increasing the performance of the proposed classifier when a limited number of epochs is considered. Moreover, the SCG methods avoid the computation of a full Hessian matrix, which reduces computational complexity when second-order information training methods are adopted;
- •
Performance analyses in terms of accuracy, mean squared error (MSE), convergence speed, number of fuzzy rules, and well-established classification metrics are presented based on datasets provided by UCI Machine Learning Repository [16]. We also present comparative results for the proposed extension and T1-FLS using OvA decomposition strategy;
- •
A simple form to compute using only matrix operations is presented to clarify the implementation of the SCGR training method for T1-FLSMO or T1-FLS.
The remainder of this paper is organized as follows. Section 2 presents the problem formulation. Section 3 addresses T1-FLSMO, as well as the SCG and SCGR training methods. Section 4 focuses on the analyses of experimental results. Section 5 states our main conclusions regarding the proposed extension; Appendix A outlines the deduction of using the differential operator and Appendix B presents the computation of using the differential operator based solely on matrix operations for T1-FLSMO.
Section snippets
Problem statement
A classification problem can be formulated as mapping between a vector and a label, where P is the number of features . The input space is divided into “decision regions”, whose boundaries are called “decision boundaries” or “decision surfaces” [17]. In this context, each decision region is assigned to one class. An FLS classifier maps an input vector to decision regions by using IF-THEN rules. When adopting a T1-FLS BC for handling classification problem proposed in [18] and
The proposed type-1 singleton fuzzy logic system multiple output
The proposed extension of T1-FLS is called T1-FLSMO and its inference block is presented in Fig. 1.
The multiple outputs of T1-FLSMO are related to deffuzifier block, as shown in Fig. 1. One can increase number of outputs of T1-FLS using height defuzzification, which replaces each rule in an output fuzzy set with a singleton () at the point having the maximum membership value in that output set. Next, by calculating the centroid of the T1 set comprised of these singletons, one can create a
Experimental results
The performance analyses presented in this section are based on datasets provided by UCI Machine Learning Repository [16]. Information regarding these datasets is presented in Table 1. For the sake of comparison, we implemented the T1-FLS using the OvA decomposition strategy and investigated training using the SD (SD T1-FLSOvA) and SCG (SCG T1-FLSOvA) based on the algorithm presented in Fig. 2. A T1-FLS trained using the SCGR method was not be implemented in this study because the results
Conclusion
In this paper, we introduced the T1-FLSMO model, which facilitates use of the T1-FLS for handling MCP without any decomposition strategy. To improve T1-FLSMO performance, we implemented the SCG and SCGR training methods, which respectively approximate and compute the multiplication of a Hessian matrix by directional vector (i.e., ). Furthermore, deductions of the equations for computing exactly using the differential operator were provided.
Numerical results demonstrated that T1-FLSMO
CRediT authorship contribution statement
Renan P. Finotti Amaral: Conceptualization, Methodology, Software, Validation, Formal analysis, Writing - original draft. Ivan F.M. Menezes: Resources, Writing - review & editing, Supervision. Moisés V. Ribeiro: Validation, Formal analysis, Writing - review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
This work was partially supported by the National Council for Scientific and Technological Development (CNPq – Conselho Nacional de Desenvolvimento Cientfico e Tecnolgico – Brazil). It was also financed by the Coordenao de Aperfeioamento de Pessoal de Nvel Superior – Brasil (CAPES) – Finance Code 001. The authors acknowledge the support provided by the Tecgraf Institute of Technical-Scientific Software Development of PUC-Rio (Tecgraf/PUC-Rio), Brazil. Any opinions, findings, conclusions, or
Renan P. Finotti Amaral received the B.Sc. degree in Mechanical Engineering from Federal University of Juiz de Fora (UFJF), Brazil, in 2015 and M.Sc. in Electrical Engineering from Federal University of Juiz de Fora (UFJF), Brazil, in 2017. He is a PhD student in Mechanical Engineering at Pontifical Catholic University of Rio de Janeiro (PUC-RIO), with the scholarship granted by CNPq (National Council for Scientific and Technological Development). His research interests include Thermosciences,
References (27)
- et al.
Solving multi-class problems with linguistic fuzzy rule based classification systems based on pairwise learning and preference relations
Fuzzy Sets Syst.
(2010) - et al.
Type-1 and singleton fuzzy logic system trained by a fast scaled conjugate gradient methods for dealing with binary classification problems
Neurocomputing
(2019) A scaled conjugate gradient algorithm for fast supervised learning
Neural Netw.
(1993)- et al.
The application of multiclass svm to the detection of knee pathologies using kinetic data: a preliminary study
- et al.
Comparison of multiclass svm classification techniques in an audio surveillance application under mismatched conditions
- Y. Lin, F. Lv, S. Zhu, M. Yang, T. Cour, K. Yu, L. Cao, T. Huang, Large-scale image classification: fast feature...
- B. Zhao, F. Li, E. P. Xing, Large-scale category structure aware image categorization, in: Advances in Neural...
- et al.
Good practice in large-scale learning for image classification
IEEE Trans. Pattern Anal. Mach. Intell.
(2014) - P.H.S. Calderano, M.G.C. Ribeiro, R.P.F. Amaral, M.M.B.R. Vellasco, R. Tanscheit, E.P. de Aguiar, An enhanced aircraft...
- R.A. Campos, R.P.F. Amaral, N. Soares, L.G. da Fonseca, M.L. Lagares Júnior, E.P. de Aguiar, A new model to distinguish...
Efficient classification for multiclass problems using modular neural networks
IEEE Trans. Neural Netw.
Reducing multiclass to binary: a unifying approach for margin classifiers
J. Mach. Learn. Res.
Cited by (4)
Energy reliability in macro base stations: A feasible solution based on a type-1 Mamdani fuzzy system
2021, Electric Power Systems ResearchCitation Excerpt :Also, this fuzzy system relies on the simplicity of its implementation and updating in the processor of the DataConcentrator device. The idea behind multiple-output fuzzy system was formerly introduced in [40]. In this contribution, it is investigated together with the Center of Gravity defuzzifier and specialist-based rule design to avoid the necessity of using the training procedure suggested in [40].
Type-1 and singleton fuzzy logic system binary classifier trained by BFGS optimization method
2023, Fuzzy Optimization and Decision MakingEvaluation of the compressive strength and Cl<sup>−</sup> content of the blast furnace slag-soda sludge-based cementitious material using machine-learning approaches
2022, Clean Technologies and Environmental Policy
Renan P. Finotti Amaral received the B.Sc. degree in Mechanical Engineering from Federal University of Juiz de Fora (UFJF), Brazil, in 2015 and M.Sc. in Electrical Engineering from Federal University of Juiz de Fora (UFJF), Brazil, in 2017. He is a PhD student in Mechanical Engineering at Pontifical Catholic University of Rio de Janeiro (PUC-RIO), with the scholarship granted by CNPq (National Council for Scientific and Technological Development). His research interests include Thermosciences, Computational Fluid Dynamics, Computational Intelligence methods and applications, Optimization methods, and Digital Signal Processing.
Ivan F. M. Menezes received the B.Sc. degree in Civil Engineering from Federal University of Pernambuco (UFPE), Brazil, in 1986, M.Sc. in Civil Engineering from Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Brazil, in 1990, and D.Sc. in Civil Engineering from Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Brazil, in 1995. He is an Assistant Professor in the Mechanical Engineering Department at Pontifical Catholic University of Rio de Janeiro (PUC Rio). His research interests include Computational Mechanics, Structural Optimization, and Numerical Analysis.
Moisés V. Ribeiro received the B.Sc. degree in Electrical Engineering from the Federal University of Juiz de Fora (UFJF), MG, Brazil, and M.Sc. and D.Sc. degrees in Electrical Engineering from the University of Campinas, SP, Brazil, in 1999, 2001 and 2005, respectively. He was a Visiting Scholar at University of California in Santa Barbara, CA, USA, in 2004, Visiting Professor (2005-2007) and Assistant Professor (2007-2015) at UFJF. Since 2015, he has been an Associate Professor at UFJF. He co-founded Smarti9 LTD. and Wari LTD. in 2012 and 2015, respectively. His research interests includes signal processing, digital communication, power line communication, smart grids, internet of things and smart city. In these fields, he has authored over 172 peer reviewed papers, 9 book chapters, and filed 13 patents. He was the General Chair of the 2010 IEEE ISPLC, 2013 IWSGC, SBrT 2015, and a Guest Co-Editor for Special Issues in the EURASIP Journal on Advances in Signal Processing and EURASIP Journal of Electrical and Computer Engineering. He had served as the Secretary of the IEEE ComSoc TC-PLC. He was the recipient of Fulbright Visiting Professorship at Stanford University, Stanford, CA, USA, in 2011, and at Princeton University, Princeton, NJ, USA, in 2012. He was awarded Student Awards from 2001 IEEE IECON and 2003 IEEE ISIE.