Skip to main content
SpringerLink
Log in

Apache Spark Implementation of Whale Optimization Algorithm

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

Population-based meta-heuristic algorithms are among the dominant algorithms used to solve challenging real world problems in diverse fields. Whale Optimization Algorithm (WOA) is a recent swarm intelligence meta-heuristic algorithm based on the bubble-net feeding behavior of humpback whales. Despite its capability to solve complex optimization problems, WOA requires enormous amount of computations when solving large size problems. This work proposes Spark-WOA, a distributed implementation of WOA on Apache Spark platform to enhance its performance and reduce computational complexity. The proposed algorithm exploits in-memory computations and broadcast features of Apache Spark to provide better performance and scalability. Details of the proposed algorithm are presented and its performance as compared to a recent Apache Hadoop implementation is discussed. Experimental results demonstrated the superiority of the proposed implementation in terms of both speed and scalability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Abd El Aziz, M., Ewees, A.A., Hassanien, A.E.: Whale optimization algorithm and moth-flame optimization for multilevel thresholding image segmentation. Expert Syst. Appl. 83, 242–256 (2017)

    Article  Google Scholar 

  2. Alnafessah, A., Casale, G.: Artificial neural networks based techniques for anomaly detection in apache spark. Clust. Comput. 23, 1361–1362 (2020)

    Article  Google Scholar 

  3. Barba-Gonzaléz, C., García-Nieto, J., Nebro, A.J., Aldana-Montes, J.F.: Multi-objective big data optimization with jmetal and spark. In: Proceedings of the International Conference on Evolutionary Multi-Criterion Optimization, pp. 16–30. Springer (2017)

  4. Chen, H., Hu, Z., Han, L., Hou, Q., Ye, Z., Yuan, J., Zeng, J.: A spark-based distributed whale optimization algorithm for feature selection. In: Proceedings of the 2019 10th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS), vol. 1, pp. 70–74. IEEE (2019)

  5. Cheraghchi, F., Iranzad, A., Raahemi, B.: Subspace selection in high-dimensional big data using genetic algorithm in apache spark. In: Proceedings of the Second International Conference on Internet of things, Data and Cloud Computing, pp. 1–7 (2017)

  6. Dorigo, M., Birattari, M., Stutzle, T.: Ant colony optimization. IEEE Comput. Intell. Mag. 1(4), 28–39 (2006)

    Article  Google Scholar 

  7. García, J., Altimiras, F., Peña, A., Astorga, G., Peredo, O.: A binary cuckoo search big data algorithm applied to large-scale crew scheduling problems. Complexity (2018). https://doi.org/10.1155/2018/8395193

    Article  Google Scholar 

  8. Gharehchopogh, F.S., Gholizadeh, H.: A comprehensive survey: whale optimization algorithm and its applications. Swarm Evol. Comput. 48, 1–24 (2019)

    Article  Google Scholar 

  9. He, F., Wei, P.: Research on comprehensive point of interest (poi) recommendation based on spark. Clust. Comput. 22(4), 9049–9057 (2019)

    Article  Google Scholar 

  10. He, Z., Peng, H., Chen, J., Deng, C., Wu, Z.: A spark-based differential evolution with grouping topology model for large-scale global optimization. Clust. Comput. (2020). https://doi.org/10.1007/s10586-020-03124-z

    Article  Google Scholar 

  11. Holland, J.H.: Genetic algorithms. Sci. Am. 267(1), 66–73 (1992)

    Article  Google Scholar 

  12. Huang, X., Li, C., Chen, H., An, D.: Task scheduling in cloud computing using particle swarm optimization with time varying inertia weight strategies. Clust. Comput. 23, 1137–1147 (2020)

    Article  Google Scholar 

  13. Ilango, S.S., Vimal, S., Kaliappan, M., Subbulakshmi, P.: Optimization using artificial bee colony based clustering approach for big data. Clust. Comput. 22(5), 12169–12177 (2019)

    Article  Google Scholar 

  14. Karaboga, D., Basturk, B.: A powerful and efficient algorithm for numerical function optimization: artificial bee colony (abc) algorithm. J. Global Optim. 39(3), 459–471 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  15. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of ICNN’95-International Conference on Neural Networks, vol. 4, pp. 1942–1948. IEEE (1995)

  16. Khalil, Y., Alshayeji, M., Ahmad, I.: Distributed whale optimization algorithm based on mapreduce. Concurr. Comput. Pract. Exp. 31(1), e4872 (2019)

    Article  Google Scholar 

  17. Kong, F., Lin, X.: The method and application of big data mining for mobile trajectory of taxi based on mapreduce. Clust. Comput. 22(5), 11435–11442 (2019)

    Article  Google Scholar 

  18. Lämmel, R.: Google’s mapreduce programming model—revisited. Sci. Comput. Programm. 70(1), 1–30 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  19. Li, B., Li, J., Tang, K., Yao, X.: Many-objective evolutionary algorithms: a survey. ACM Comput. Surv. (CSUR) 48(1), 1–35 (2015)

    Article  Google Scholar 

  20. Li, C., Wen, T., Dong, H., Wu, Q., Zhang, Z.: Implementation of parallel multi-objective artificial bee colony algorithm based on spark platform. In: Proceedings of the 2016 11th International Conference on Computer Science & Education (ICCSE), pp. 592–597. IEEE (2016)

  21. Ling, Y., Zhou, Y., Luo, Q.: Lévy flight trajectory-based whale optimization algorithm for global optimization. IEEE Access 5, 6168–6186 (2017)

    Article  Google Scholar 

  22. Lu, H.C., Hwang, F., Huang, Y.H.: Parallel and distributed architecture of genetic algorithm on apache hadoop and spark. Appl. Soft Comput. (2020). https://doi.org/10.1016/j.asoc.2020.106497

    Article  Google Scholar 

  23. Luo, X., Fu, X.: Configuration optimization method of hadoop system performance based on genetic simulated annealing algorithm. Clust. Comput. 22(4), 8965–8973 (2019)

    Article  MathSciNet  Google Scholar 

  24. Małysiak-Mrozek, B., Baron, T., Mrozek, D.: Spark-idpp: high-throughput and scalable prediction of intrinsically disordered protein regions with spark clusters on the cloud. Clust. Comput. 22(2), 487–508 (2019)

    Article  Google Scholar 

  25. Manogaran, G., Lopez, D.: A gaussian process based big data processing framework in cluster computing environment. Clust. Comput. 21(1), 189–204 (2018)

    Article  Google Scholar 

  26. Mirjalili, S., Lewis, A.: The whale optimization algorithm. Adv. Eng. Softw. 95, 51–67 (2016)

    Article  Google Scholar 

  27. Mirjalili, S., Mirjalili, S.M., Lewis, A.: Grey wolf optimizer. Adv. Eng. Softw. 69, 46–61 (2014)

    Article  Google Scholar 

  28. OpenMP Architecture Review Board: OpenMP application program interface version 3.0 (2008). http://www.openmp.org/mp-documents/spec30.pdf

  29. Pham, Q.V., Mirjalili, S., Kumar, N., Alazab, M., Hwang, W.J.: Whale optimization algorithm with applications to resource allocation in wireless networks. IEEE Trans. Veh. Technol. 69(4), 4285–4297 (2020)

    Article  Google Scholar 

  30. Prakash, D.B., Lakshminarayana, C.: Optimal siting of capacitors in radial distribution network using whale optimization algorithm. Alex. Eng. J. 56(4), 499–509 (2017)

    Article  Google Scholar 

  31. Ramírez-Gallego, S., García, S., Benítez, J.M., Herrera, F.: A distributed evolutionary multivariate discretizer for big data processing on apache spark. Swarm Evol. Comput. 38, 240–250 (2018)

    Article  Google Scholar 

  32. Sauber, A.M., Nasef, M.M., Houssein, E.H., Hassanien, A.E.: Parallel whale optimization algorithm for solving constrained and unconstrained optimization problems. arXiv preprint arXiv:1807.09217 (2018)

  33. Sherar, M., Zulkernine, F.: Particle swarm optimization for large-scale clustering on apache spark. In: Proceedings of the 2017 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1–8. IEEE (2017)

  34. Sunderam, V.S.: PVM: a framework for parallel distributed computing. Concurr. Pract. Exp. 2(4), 315–339 (1990)

    Article  Google Scholar 

  35. Touma, H.J.: Study of the economic dispatch problem on ieee 30-bus system using whale optimization algorithm. Int. J. Eng. Technol. Sci. (IJETS) 5(1), 11–18 (2016)

    Google Scholar 

  36. Watkins, W.A., Schevill, W.E.: Aerial observation of feeding behavior in four baleen whales: eubalaena glacialis, balaenoptera borealis, megaptera novaeangliae, and balaenoptera physalus. J. Mammal. 60(1), 155–163 (1979)

    Article  Google Scholar 

  37. Wen, T., Liu, H., Lin, L., Wang, B., Hou, J., Huang, C., Pan, T., Du, Y.: Multiswarm artificial bee colony algorithm based on spark cloud computing platform for medical image registration. Comput. Methods Progr. Biomed. (2020). https://doi.org/10.1016/j.cmpb.2020.105432

    Article  Google Scholar 

  38. Xiong, F., Gong, P., Jin, P., Fan, J.: Supply chain scheduling optimization based on genetic particle swarm optimization algorithm. Clust. Comput. 22(6), 14767–14775 (2019)

    Article  Google Scholar 

  39. Yang, X.S., Deb, S.: Cuckoo search via lévy flights. In: Proceedings of the 2009 World congress on nature & biologically inspired computing (NaBIC), pp. 210–214. IEEE (2009)

  40. Yang, X.S., He, X.: Nature-inspired optimization algorithms in engineering: overview and applications. In: Yang, X.-S. (ed.) Nature-Inspired Computation in Engineering, pp. 1–20. Springer, New York (2016)

    Chapter  Google Scholar 

  41. Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I., et al.: Spark: custer computing with working sets. HotCloud 10(10–10), 95 (2010)

    Google Scholar 

  42. Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauly, M., Franklin, M.J., Shenker, S., Stoica, I.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Presented as part of the 9th {USENIX} Symposium on Networked Systems Design and Implementation {NSDI}, vol. 12, pp. 15–28 (2012)

  43. Zaharia, M., Xin, R.S., Wendell, P., Das, T., Armbrust, M., Dave, A., Meng, X., Rosen, J., Venkataraman, S., Franklin, M.J., et al.: Apache spark: a unified engine for big data processing. Commun. ACM 59(11), 56–65 (2016)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Engineering Department, Kuwait University, Kuwait, Kuwait

    Maryam AlJame, Imtiaz Ahmad & Mohammad Alfailakawi

Authors
  1. Maryam AlJame
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Imtiaz Ahmad
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohammad Alfailakawi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohammad Alfailakawi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendices

All benchmarks used in this work are mathematically described in Table 1 with their main characteristics highlighted. Graphical plots for these functions are also shown in Fig. 10.

Table 1 Benchmark functions description
Full size table
Fig. 10
figure 10

Benchmarks 2D plots

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

AlJame, M., Ahmad, I. & Alfailakawi, M. Apache Spark Implementation of Whale Optimization Algorithm. Cluster Comput 23, 2021–2034 (2020). https://doi.org/10.1007/s10586-020-03162-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-020-03162-7

Keywords

Search

Navigation