Abstract
Classification is a data mining task the goal of which is to learn a model, from a training dataset, that can predict the class of a new data instance, while clustering aims to discover natural instance-groupings within a given dataset. Learning cluster-based classification systems involves partitioning a training set into data subsets (clusters) and building a local classification model for each data cluster. The class of a new instance is predicted by first assigning the instance to its nearest cluster and then using that cluster’s local classification model to predict the instance’s class. In this paper, we present an ant colony optimization (ACO) approach to building cluster-based classification systems. Our ACO approach optimizes the number of clusters, the positioning of the clusters, and the choice of classification algorithm to use as the local classifier for each cluster. We also present an ensemble approach that allows the system to decide on the class of a given instance by considering the predictions of all local classifiers, employing a weighted voting mechanism based on the fuzzy degree of membership in each cluster. Our experimental evaluation employs five widely used classification algorithms: naïve Bayes, nearest neighbour, Ripper, C4.5, and support vector machines, and results are reported on a suite of 54 popular UCI benchmark datasets.
Similar content being viewed by others
References
Abdallah, Z. S., Gaber, M., Srinivasan, B., & Krishnaswamy, S. (2012). CBARS: Cluster based classification for activity recognition systems. In A. Hassanien, A. Salem, R. Ramadan & T. Kim (Eds.), Advanced machine learning technologies and applications (Proceedings AMLTA-2012), Communications in Computer and Information Science (CCIS) (Vol. 322, pp. 82–91). Berlin: Springer.
Abdelbar, A. M., & Salama, K. M. (2016). Clustering with the ACO\({_{\mathbb{R}}}\) algorithm. In: M. Dorigo, M. Birattari, X. Li, M. López-Ibáñez, K. Ohkura, C. Pinciroli & T. Stützle (Eds.), Swarm intelligence (Proceedings ANTS-2016), Lecture Notes in Computer Science (Vol. 9882, pp. 291–292). Berlin: Springer.
Arthur, D., & Vassilvitskii, S. (2007). k-means\(++\): The advantages of careful seeding. In Proceedings ACM-SIAM Symposium on Discrete Algorithms (SODA-2007) (pp. 1027–1035). Philadelphia, PA: Society for Industrial and Applied Mathematics.
Ashok, L., & Messinger, D. (2012). A spectral image clustering algorithm based on ant colony optimization. In Proceedings SPIE 8390, Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XVIII (p. 83901P). SPIE Press.
Asuncion, A., & Newman, D. (2007). UCI Machine Learning Repository. http://www.ics.uci.edu/~mlearn/MLRepository.html.
Bezdek, J., Ehrlich, R., & Full, W. (1984). FCM: The fuzzy \(c\)-means clustering algorithm. Computers & Geosciences, 10(2–3), 191–203.
Bishop, C. M. (2007). Pattern recognition and machine learning. Berlin: Springer.
Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123–140.
Dempster, A., Laird, N., & Rubin, D. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B (Methodological), 39(1), 1–8.
Dorigo, M., & Stützle, T. (2004). Ant colony optimization. Cambridge: MIT Press.
Franca, F., Coelho, G., & Zuben, F. (2008). bicACO: An ant colony inspired biclustering algorithm. In M. Dorigo, M. Birattari, C. Blum, M. Clerc, T. Stützle & A. Winfield (Eds.), Ant colony optimization and swarm intelligence (Proceedings ANTS-2008), Lecture Notes in Computer Science (Vol. 5217, pp. 401–402). Berlin: Springer.
Freund, Y., & Schapire, R. (1997). Decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1), 119–139.
Gan, G., Ma, C., & Wu, J. (2007). Data clustering: Theory, algorithms, and applications. Philadelphia, PA: SIAM Press.
Han, J., & Kamber, M. (2000). Data mining: Concepts and techniques. San Francisco, CA: Morgan Kaufmann.
Jafar, M., & Sivakumar, R. (2010). Ant-based clustering algorithms: A brief survey. International Journal of Computer Theory and Engineering, 2, 787–796.
Kao, Y., & Cheng, K. (2016). An ACO-based clustering algorithm. In: M. Dorigo, L. M. Gambardella, M. Birattari, A. Martinoli, R. Poli & T. Stützle (Eds.), Swarm intelligence (Proceedings ANTS-2006), Lecture Notes in Computer Science (Vol. 4150, pp 210–221). Berlin: Springer.
Kaufman, L., & Rousseeuw, P. (1987). Clustering by means of medoids. In Y. Dodge (Ed.), Statistical data analysis based on the L1 norm and related methods (pp. 405–416). Amsterdam: North-Holland.
Liao, T., Socha, K., Montes de Oca, M., Stützle, T., & Dorigo, M. (2014). Ant colony optimization for mixed-variable optimization problems. IEEE Transactions on Evolutionary Computation, 18(4), 503–518.
Liu, X. Y., & Fu, H. (2010). An effective clustering algorithm with ant colony. Journal of Computers, 5, 598–605.
Liu, Y., Li, Z., Xiong, H., Gao, X., & Wu, J. (2010). Understanding of internal clustering validation measures. In Proceedings IEEE conference on data mining (pp. 911–916). IEEE Press.
MacQueen, J. (1967). Some methods of classification and analysis of multivariate observations. In Proceedings fifth Berkeley symposium on mathematical statistics and probability (Vol. 1, pp. 281–297). University of California Press.
Martens, D., De Backer, M., Haesen, R., Vanthienen, J., Snoeck, M., & Baesens, B. (2007). Classification with ant colony optimization. IEEE Transactions on Evolutionary Computation, 11(5), 651–665.
Mehrotra, K., Ozgencil, N., & McCracken, N. (2007). Squeezing the last drop: Cluster-based classification algorithm. Statistics & Probability Letters, 77, 1288–1299.
Menendez, H., Otero, F., & Camacho, D.(2014a). MACOC: A medoid-based ACO clustering algorithm. In: M. Dorigo, M. Birattari, S. Garnier, H. Hamann, M. Montes de Oca, C. Solnon & T. Stützle (Eds.,) Swarm Intelligence (Proceedings ANTS-2014), Lecture Notes in Computer Science (Vol. 8667, pp. 122–133). Berlin: Springer.
Menendez, H., Otero, F., & Camacho, D. (2014b). SACOC: A spectral-based ACO clustering algorithm. In Proceedings eighth international symposium on intelligent distributed computing, Studies in Computational Intelligence (Vol. 570, pp. 185–194). Berlin: Springer
Otero, F. E., Freitas, A. A., & Johnson, C. G. (2009). Handling continuous attributes in ant colony classification algorithms. In IEEE Symposium on Computational Intelligence in Data Mining (CIDM 2009) (pp 225–231). IEEE Press.
Otero, F. E., Freitas, A. A., & Johnson, C. G. (2012). Inducing decision trees with an ant colony optimization algorithm. Applied Soft Computing, 12(11), 3615–3626.
Otero, F. E., Freitas, A. A., & Johnson, C. (2013). A new sequential covering strategy for inducing classification rules with ant colony algorithms. IEEE Transactions on Evolutionary Computation, 17(1), 64–74.
Ozgencil, N. E. (2007). Cluster based classification for semantic role labeling. Ph.D. thesis, Syracuse University, Syracuse, NY, USA.
Parpinelli, R. S., Lopes, H. S., & Freitas, A. A. (2002). Data mining with an ant colony optimization algorithm. IEEE Transactions on Evolutionary Computation, 6(4), 321–332.
Rokach, L. (2010). Ensemble-based classifiers. Artificial Intelligence Review, 33(1–2), 1–39.
Salama, K. M., & Abdelbar, A. M. (2015). Learning neural network structures with ant colony algorithms. Swarm Intelligence, 9(4), 229–265.
Salama, K. M., & Abdelbar, A. M. (2016). Using ant colony optimization to build cluster-based classification systems. In M. Dorigo, M. Birattari, X. Li, M. López-Ibáñez, K. Ohkura, C. Pinciroli & T. Stützle (Eds.), Swarm Intelligence (Proceedings ANTS-2016), Lecture Notes in Computer Science (Vol. 9882, pp. 210–221). Berlin: Springer.
Salama, K. M., & Freitas, A. A. (2013a). Clustering-based Bayesian multi-net classifier construction with ant colony optimization. In IEEE Congress on Evolutionary Computation (IEEE CEC) (pp. 3079–3086). IEEE Press.
Salama, K. M., & Freitas, A. A. (2013b). Learning Bayesian network classifiers using ant colony optimization. Swarm Intelligence, 7(2–3), 229–254.
Salama, K. M., & Freitas, A. A. (2014a). ABC-Miner\(+\): Constructing Markov blanket classifiers with ant colony algorithms. Memetic Computing, 6(3), 183–206.
Salama, K. M., & Freitas, A. A. (2014b). Classification with cluster-based Bayesian multi-nets using ant colony optimization. Swarm and Evolutionary Computation, 18, 54–70.
Salama, K. M., & Freitas, A. A. (2015). Ant colony algorithms for constructing Bayesian multi-net classifiers. Intelligent Data Analysis, 19(2), 233–257.
Salama, K. M., & Otero, F. E. (2014). Learning multi-tree classification models with ant colony optimization. In Proceedings sixth international conference on evolutionary computation theory and applications (ECTA-2014), Lecture Notes in Computer Science (pp. 38–48). Berlin: Springer.
Salama, K. M., Abdelbar, A. M., & Freitas, A. A. (2011). Multiple pheromone types and other extensions to the Ant-Miner classification rule discovery algorithm. Swarm Intelligence, 5(3–4), 149–182.
Salama, K. M., Abdelbar, A. M., & Anwar, I. M. (2016). Data reduction for classification with ant colony optimization. Intelligent Data Analysis, 20(5), 1021–1059.
Salama, K. M., Abdelbar, A. M., Helal, A. Z., & Freitas, A. A. (2017). Instance-based classification with ant colony optimization. Intelligent Data Analysis (to appear).
Schapire, R. (1990). The strength of weak learnability. Machine Learning, 5(2), 197–227.
Shelokar, P. S., Jayaraman, V. K., & Kulkarni, B. D. (2004). An ant colony approach for clustering. Analytica Chimica Acta, 509(2), 187–195.
Socha, K., & Blum, C. (2007). An ant colony optimization algorithm for continuous optimization: Application to feed-forward neural network training. Neural Computing & Applications, 16, 235–247.
Socha, K., & Dorigo, M. (2008). Ant colony optimization for continuous domains. European Journal of Operational Research, 185, 1155–1173.
Tan, P. N., Steinbach, M., & Kumar, V. (2005). Introduction to data mining. Reading: Addison Wesley.
Witten, I. H., & Frank, E. (2010). Data mining: Practical machine learning tools and techniques. San Francisco, CA: Morgan Kaufmann.
Woniak, M., Graña, M., & Corchado, E. (2014). A survey of multiple classifier systems as hybrid systems. Information Fusion, 16, 3–17.
Xu, R., & Wunsch, D. (2009). Clustering. Piscataway, NJ: IEEE Press.
Yen, S. J., & Lee, Y. S. (2009). Cluster-based under-sampling approaches for imbalanced data distributions. Expert Systems with Applications, 36, 5718–5727.
Zhang, X., & Xiao, W. (2012). Clustering based two-stage text classification requiring minimal training data. Computer Science and Information Systems, 9(4), 1627–1643.
Acknowledgements
The partial support of a grant from the Brandon University Research Council (BURC) is gratefully acknowledged. The authors would like to thank the anonymous reviewers and the guest editor for their insightful comments which substantially improved the paper.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Salama, K.M., Abdelbar, A.M. Learning cluster-based classification systems with ant colony optimization algorithms. Swarm Intell 11, 211–242 (2017). https://doi.org/10.1007/s11721-017-0138-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11721-017-0138-5