Abstract
Weather forecasting, as an important and indispensable procedure in people’s daily lives, evaluates the alteration happening in the current condition of the atmosphere. Big data analytics is the process of analyzing big data to extract the concealed patterns and applicable information that can yield better results. Nowadays, several parts of society are interested in big data, and the meteorological institute is not excluded. Therefore, big data analytics will give better results in weather forecasting and will help forecasters to forecast weather more accurately. In order to achieve this goal and to recommend favorable solutions, several big data techniques and technologies have been suggested to manage and analyze the huge volume of weather data from different resources. By employing big data analytics in weather forecasting, the challenges related to traditional data management techniques and technology can be solved. This paper tenders a systematic literature review method for big data analytic approaches in weather forecasting (published between 2014 and August 2020). A feasible taxonomy of the current reviewed papers is proposed as technique-based, technology-based, and hybrid approaches. Moreover, this paper presents a comparison of the aforementioned categories regarding accuracy, scalability, execution time, and other Quality of Service factors. The types of algorithms, measurement environments, modeling tools, and the advantages and disadvantages per paper are extracted. In addition, open issues and future trends are debated.
Similar content being viewed by others
Change history
20 July 2021
A Correction to this paper has been published: https://doi.org/10.1007/s11831-021-09630-6
References
Xiao Z, Liu B, Liu H, Zhang D (2012) Progress in climate prediction and weather forecast operations in China. Adv Atmos Sci 29(5):943–957
Bengtsson L (1980) The weather forecast. Pure Appl Geophys 119(3):515–537
Kan L, Yu-Shu L (2005) A rough set based fuzzy neural network algorithm for weather prediction. In: 2005 International conference on machine learning and cybernetics, vol 3. pp 1888–1892
Kan L, Yu-Shu L (2002) Fuzzy case-based reasoning: weather prediction. In: Proceedings of the international conference on machine learning and cybernetics, vol 1. pp 107–110
Weiguo X (2010) The weather prediction method based on artificial immune system. In: 2010 International forum on information technology and applications, vol 2. pp 386–389
Haupt SE, Cowie J, Linden S, McCandless T, Kosovic B, Alessandrini S (2018) Machine learning for applied weather prediction. In: 2018 IEEE 14th international conference on e-science (e-Science). pp 276–277
Chung CYC, Kumar VR (1993) Knowledge acquisition using a neural network for a weather forecasting knowledge-based system. Neural Comput Appl 1(3):215–223
Pandey AK, Agrawal CP, Agrawal M (2017) A hadoop based weather prediction model for classification of weather data. In: 2017 Second international conference on electrical, computer and communication technologies (ICECCT). pp 1–5
Tsai C-W, Lai C-F, Chao H-C, Vasilakos AV (2015) Big data analytics: a survey. J Big Data 2(1):21
Rodríguez-Mazahua L, Rodríguez-Enríquez C-A, Sánchez-Cervantes JL, Cervantes J, García-Alcaraz JL, Alor-Hernández G (2016) A general perspective of big data: applications, tools, challenges and trends. J Supercomput 72(8):3073–3113
Talia D (2013) Clouds for scalable big data analytics. Computer 46(5):98–101
Selvaraj P, Marudappa P (2018) A survey of predictive analytics using big data with data mining. Int J Bioinf Res Appl 14:269
Sharma S, Mangat V (2015) Technology and trends to handle big data: survey. In: 2015 Fifth international conference on advanced computing and communication technologies. pp 266–271
Jain H, Jain R (2017) Big data in weather forecasting: applications and challenges. In: 2017 International conference on big data analytics and computational intelligence (ICBDAC). pp 138–142
Reddy PC, Babu AS (2017) Survey on weather prediction using big data analystics. In: 2017 Second international conference on electrical, computer and communication technologies (ICECCT). pp 1–6
Bendre MR, Thool RC, Thool VR (2015) Big data in precision agriculture: weather forecasting for future farming. In: 2015 1st international conference on next generation computing technologies (NGCT). pp 744–750
Mittal S, Sangwan OP (2019) Big data analytics using data mining techniques: a survey. In: Advanced informatics for computing research, Singapore. Springer Singapore, pp 264–273
Leu J-S, Su K-W, Chen C-T (2014) Ambient mesoscale weather forecasting system featuring mobile augmented reality. Multimed Tools Appl 72(2):1585–1609
Corne D, Dissanayake M, Peacock A, Galloway S, Owens E (2014) Accurate localized short term weather prediction for renewables planning. In: 2014 IEEE symposium on computational intelligence applications in smart grid (CIASG). pp 1–8
Roudier P et al (2014) The role of climate forecasts in smallholder agriculture: lessons from participatory research in two communities in Senegal. Clim Risk Mana 2:42–55
Li J, Xu L, Tang L, Wang S, Li L (2018) Big data in tourism research: a literature review. Tour Manag 68:301–323
Scott D, Lemieux C (2010) Weather and climate information for tourism. Procedia Environ Sci 1:146–183
Hazyuk I, Ghiaus C, Penhouet D (2012) Optimal temperature control of intermittently heated buildings using Model Predictive Control: Part I—Building modeling. Build Environ 51:379–387
Enríquez R, Jiménez MJ, Heras MdR (2016) Solar forecasting requirements for buildings MPC. Energy Procedia 91:1024–1032
Smith DA, Sherry L (2008) Decision support tool for predicting aircraft arrival rates from weather forecasts. In: 2008 Integrated communications, navigation and surveillance conference. pp 1–12
Zhang B, Tang L, Roemer M (2018) Probabilistic planning and risk evaluation based on ensemble weather forecasting. IEEE Trans Autom Sci Eng 15(2):556–566
Braman LM, van Aalst MK, Mason SJ, Suarez P, Ait-Chellouche Y, Tall A (2013) Climate forecasts in disaster management: red cross flood operations in West Africa, 2008. Disasters 37(1):144–164
Akhand MH (2003) Disaster management and cyclone warning system in Bangladesh. In: Zschau J, Küppers A (eds) Early warning systems for natural disaster reduction. Springer, Berlin, pp 49–64
Chen C, Duan S, Cai T, Liu B (2011) Online 24-h solar power forecasting based on weather type classification using artificial neural network. Sol Energy 85(11):2856–2870
Shi J, Lee W, Liu Y, Yang Y, Wang P (2012) Forecasting power output of photovoltaic systems based on weather classification and support vector machines. IEEE Trans Ind Appl 48(3):1064–1069
Lazos D, Sproul AB, Kay M (2014) Optimisation of energy management in commercial buildings with weather forecasting inputs: a review. Renew Sustain Energy Rev 39:587–603
Casas DM, González JÁT, Rodríguez JEA, Pet JV (2009) Using data-mining for short-term rainfall forecasting. In: Distributed computing, artificial intelligence, bioinformatics, soft computing, and ambient assisted living. Springer, Berlin, pp 487–490
Katal A, Wazid M, Goudar RH (2013) Big data: issues, challenges, tools and good practices. In: 2013 Sixth international conference on contemporary computing (IC3). pp 404–409
Elgendy N, Elragal A (2014) Big data analytics: a literature review paper. In: Advances in data mining. Applications and theoretical aspects. Springer, Cham, pp 214–227
Shadroo S, Rahmani A (2018) Systematic survey of big data and data mining in internet of things. Comput Netw 139:19–47
Bazzaz Abkenar S, Mahdipour E, Jameii SM, Haghi Kashani M (2021) A hybrid classification method for Twitter spam detection based on differential evolution and random forest. Concurr Comput Pract Exp. https://doi.org/10.1002/cpe.6381
Pathak AR, Pandey M, Rautaray S (2018) Construing the big data based on taxonomy, analytics and approaches. Iran J Comput Sci 1(4):237–259
Bazzaz Abkenar S, Haghi Kashani M, Mahdipour E, Jameii SM (2021) Big data analytics meets social media: a systematic review of techniques, open issues, and future directions. Telemat Inform 57:101517
Khezr SN, Navimipour NJ (2017) MapReduce and its applications, challenges, and architecture: a comprehensive review and directions for future research. J Grid Comput 15(3):295–321
Amer A-B, Amr M, Salah H (2016) A survey on MapReduce implementations. Int J Cloud Appl Comput IJCAC 6(1):59–87
Senger H et al (2016) BSP cost and scalability analysis for MapReduce operations. Concurr Comput Pract Exp 28(8):2503–2527
Lee D, Kim JW, Maeng S (2014) Large-scale incremental processing with MapReduce. Future Gener Comput Syst 36:66–79
Idris M et al (2015) Context-aware scheduling in MapReduce: a compact review. Concurr Comput Pract Exp 27(17):5332–5349
Karimi Y, Haghi Kashani M, Akbari M, Mahdipour E (2021) Leveraging big data in smart cities: a systematic review. J Concurr Comput Pract Exp. https://doi.org/10.1002/cpe.6379
Bakratsas M, Basaras P, Katsaros D, Tassiulas L (2018) Hadoop MapReduce performance on SSDs for analyzing social networks. Big Data Res 11:1–10
Shabestari F, Rahmani AM, Navimipour NJ, Jabbehdari S (2019) A taxonomy of software-based and hardware-based approaches for energy efficiency management in the Hadoop. J Netw Comput Appl 126:162–177
Patwardhan A, Verma AK, Kumar U (2016) A survey on predictive maintenance through big data. In: Current trends in reliability, availability, maintainability and safety. Springer, Cham, pp 437–445
Yang W, Liu X, Zhang L, Yang LT (2013) Big data real-time processing based on storm. In: 2013 12th IEEE international conference on trust, security and privacy in computing and communications. pp 1784–1787
Philip-Chen CL, Zhang C-Y (2014) Data-intensive applications challenges techniques and technologies: a survey on big data. Inf Sci 275:314–347
Lee J, Hong S, Lee J-H (2014) An efficient prediction for heavy rain from big weather data using genetic algorithm. In: Presented at the proceedings of the 8th international conference on ubiquitous information management and communication, Siem Reap, Cambodia
Sahasrabuddhe DV, Jamsandekar P (2015) Data structure for representation of big data of weather forecasting: a review. Int J Comput Sci Trends Technol IJCST 3(6):48–56
Priya SB A survey on weather forecasting to predict rainfall using big data analytics
Hassani H, Silva ES (2015) Forecasting with big data: a review. Ann Data Sci 2(1):5–19
Rao N (2017) Big data and climate smart agriculture-review of current status and implications for agricultural research and innovation in India. In: Proceedings Indian National Science Academy, Forthcoming
de Freitas Viscondi G, Alves-Souza SN (2019) A systematic literature review on big data for solar photovoltaic electricity generation forecasting. Sustain Energy Technol Assess 31:54–63
Vannitsem S et al (2021) Statistical postprocessing for weather forecasts: review, challenges, and avenues in a big data world. Bull Am Meteorol Soc 102(3):E681–E699
Cook DJ, Greengold NL, Ellrodt AG, Weingarten SR (1997) The relation between systematic reviews and practice guidelines. Ann Internal Med 127(3):210–216
Haghi Kashani M, Rahmani AM, Jafari Navimipour N (2020) Quality of service-aware approaches in fog computing. Int J Commun Syst 33(8):e4340
Rahimi M, Songhorabadi M, Haghi Kashani M (2020) Fog-based smart homes: a systematic review. J Netw Comput Appl 153:102531
Bazzaz Abkenar S, Haghi Kashani M, Akbari M, Mahdipour E (2020) Twitter spam detection: a systematic review. arXiv preprint arXiv:2011.14754.
Songhorabadi M, Rahimi M, Farid AMM, Kashani MH (2020) Fog computing approaches in smart cities: a state-of-the-art review. arXiv preprint arXiv:2011.14732
Kashani MH, Ahmadzadeh A, Mahdipour E (2020) Load balancing mechanisms in fog computing: a systematic review. arXiv preprint arXiv:2011.14706
Brereton P, Kitchenham BA, Budgen D, Turner M, Khalil M (2007) Lessons from applying the systematic literature review process within the software engineering domain. J Syst Softw 80(4):571–583
Sheikh Sofla M, Haghi Kashani M, Mahdipour E, Faghih Mirzaee R (2021) Towards effective offloading mechanisms in fog computing: a systematic survey. Multimed Tools Appl
Haghi Kashani M, Madanipour M, Nikravan M, Asghari P, Mahdipour E (2021) A systematic review of IoT in healthcare: applications, techniques, and trends. J Netw Comput Appl
Cheng Y, Zheng Z, Wang J, Yang L, Wan S (2019) Attribute reduction based on genetic algorithm for the coevolution of meteorological data in the industrial internet of things. Wirel Commun Mob Comput 2019:8
Cramer S, Kampouridis M, Freitas A (2016) A genetic decomposition algorithm for predicting rainfall within financial weather derivatives. In: Presented at the proceedings of the genetic and evolutionary computation conference 2016, Denver, Colorado, USA
Pooja SB, Siva-Balan RV, Anisha M, Muthukumaran MS, Jothikumar R (2020) Techniques Tanimoto correlated feature selection system and hybridization of clustering and boosting ensemble classification of remote sensed big data for weather forecasting. Comput Commun 151:266–274
Kvinge H, Farnell E, Kirby M, Peterson C (2018) Monitoring the shape of weather, soundscapes, and dynamical systems: a new statistic for dimension-driven data analysis on large datasets. In: 2018 IEEE international conference on big data (big data). pp 1045–1051
Buszta A, Mazurkiewicz J (2015) Climate changes prediction system based on weather big data visualisation. In: Theory and engineering of complex systems and dependability. Springer, Cham, pp 75–86
Rasel RI, Sultana N, Meesad P (2018) An application of data mining and machine learning for weather forecasting. In: Recent advances in information and communication technology 2017. Springer, Cham, pp 169–178
Mahmood MR, Patra RK, Raja R, Sinha GR (2019) A novel approach for weather prediction using forecasting analysis and data mining techniques. In: Innovations in electronics and communication engineering. Springer, Singapore, pp 479–489
Azimi R, Ghofrani M, Ghayekhloo M (2016) A hybrid wind power forecasting model based on data mining and wavelets analysis. Energy Convers Manag 127:208–225
Doreswamy IG, Manjunatha BR (2018) Multi-label classification of big NCDC weather data using deep learning model. In: Soft computing systems. Springer, Singapore, pp 232–241
Venkatachalapathy K, Kamaleshwar T, Sundaranarayana D, Prakash VO (2016) An effective framework with N-client transfer dataset for weather prediction using data mining techniques. In: Presented at the proceedings of the international conference on informatics and analytics, Pondicherry, India
Choi C, Kim J, Kim J, Kim D, Bae Y, Kim HS (2018) Development of heavy rain damage prediction model using machine learning based on big data. Adv Meteorol 2018:11
Hubig N, Fengler P, Züfle A, Yang R, Günnemann S (2017) Detection and prediction of natural hazards using large-scale environmental data. In: Advances in spatial and temporal databases. Springer, Cham, pp 300–316
Yonekura K, Hattori H, Suzuki T (2018) Short-term local weather forecast using dense weather station by deep neural network. In: 2018 IEEE international conference on big data (big data). pp 1683–1690
Xu Q et al (2015) A short-term wind power forecasting approach with adjustment of numerical weather prediction input by data mining. IEEE Trans Sustain Energy 6(4):1283–1291
Jiang P, Dong Q (2015) A new hybrid model based on an intelligent optimization algorithm and a data denoising method to make wind speed predication. Math Probl Eng 2015:16
More PD, Nandgave S, Kadam M (2020) Weather data analytics using hadoop with map-reduce. In: ICCCE 2019. Springer, Singapore, pp 189–196
Wu H (2017) Big data management the mass weather logs. In: Smart computing and communication. Springer, Cham, pp 122–132
Ismail KA, Majid MA, Zain JM, Bakar NAA (2016) Big data prediction framework for weather temperature based on MapReduce algorithm. In: 2016 IEEE conference on open systems (ICOS). pp 13–17
Abdullahi AU, Ahmad R, Zakaria NM (2016) Big data: performance profiling of meteorological and oceanographic data on hive. In: 2016 3rd international conference on computer and information sciences (ICCOINS). pp 203–208
Oury DTM, Singh A (2018) Data analysis of weather data using hadoop technology. In: Smart computing and informatics. Springer, Singapore, pp 723–730
Manogaran G, Lopez D, Chilamkurti N (2018) In-mapper combiner based MapReduce algorithm for processing of big climate data. Future Gener Comput Syst 86:433–445
Jayanthi D, Sumathi G (2017) Weather data analysis using spark—an in-memory computing framework. In: 2017 Innovations in power and advanced computing technologies (i-PACT). pp 1–5
Palamuttam R et al (2015) SciSpark: Applying in-memory distributed computing to weather event detection and tracking. In: 2015 IEEE International conference on big data (big data). pp 2020–2026
Hassaan M, Elghandour I (2016) A real-time big data analysis framework on a CPU/GPU heterogeneous cluster: a meteorological application case study. In: 2016 IEEE/ACM 3rd international conference on big data computing applications and technologies (BDCAT). pp 168–177
Manogaran G, Lopez D (2018) Spatial cumulative sum algorithm with big data analytics for climate change detection. Comput Electr Eng 65:207–221
Madan S, Kumar P, Rawat S, Choudhury T (2018) Analysis of weather prediction using machine learning & big data. In: 2018 International conference on advances in computing and communication engineering (ICACCE). pp 259–264
Dhoot R, Agrawal S, Kumar MS (2019) Implementation and analysis of arima model and kalman filter for weather forcasting in spark computing environment. In: 2019 3rd international conference on computing and communications technologies (ICCCT). pp 105–112
Dhamodharavadhani S, Rathipriya R (2019) Region-wise rainfall prediction using mapreduce-based exponential smoothing techniques. In: Advances in big data and cloud computing. Springer, Singapore, pp 229–239
Namitha K, Jayapriya A, Kumar GS (2015) Rainfall prediction using artificial neural network on map-reduce framework. In: Presented at the proceedings of the third international symposium on women in computing and informatics, Kochi, India
Liu L, Lv J, Ma Z, Wan J, Jingjing M (2015) Toward the association rules of meteorological data mining based on cloud computing. In: Proceedings of the second international conference on mechatronics and automatic control. Springer, Cham, pp 1051–1059
Sahoo S (2017) A parallel forecasting approach using incremental K-means clustering technique. In: Computational intelligence in data mining. Springer, Singapore, pp 165–172
Fang W, Sheng VS, Wen X, Pan W (2014) Meteorological data analysis using MapReduce. Sci World J 2014:10
Hamzei M, Navimipour NJ (2018) Toward efficient service composition techniques in the internet of things. IEEE Internet Things J 5(5):3774–3787
Kumar V, Kumar D (2020) A systematic review on firefly algorithm: past, present, and future. Arch Comput Methods Eng 28(4):3269–3291
Nikravan M, Kashani MH (2007) Parallel min–max ant colony system (MMAS) for dynamic process scheduling in distributed operating systems considering load balancing. In: Proceedings of the 21st ECMS international conference on high performance computing & simulation (HPCS), Prague, Czech Republic
Kashani MH, Sarvizadeh R (2011) A novel method for task scheduling in distributed systems using max–min ant colony optimization. In: 2011 3rd international conference on advanced computer control (ICACC). IEEE, pp 422–426
Kashani MH, Zarrabi H, Javadzadeh G (2017) A new metaheuristic approach to task assignment problem in distributed systems. In: 2017 IEEE 4th international conference on knowledge-based engineering and innovation (KBEI). IEEE, pp 0673–0677
Kashani MH, Sarvizadeh R, Jameii M (2012) A new distributed systems scheduling algorithm: a swarm intelligence approach. In: Fourth international conference on machine vision (ICMV 2011): computer vision and image analysis; pattern recognition and basic technologies. International Society for Optics and Photonics
Kashani MH, Jahanshahi M (2009) A new method based on memetic algorithm for task scheduling in distributed systems. Int J Simul Syst Sci Technol 10
Niu B, Wang H (2012) Bacterial colony optimization. Discrete Dyn Nat Soc 2012:28
Kashani MH, Jahanshahi M (2009) Using simulated annealing for task scheduling in distributed systems. In: 2009 International conference on computational intelligence, modelling and simulation. pp 265–269
Dasgupta D, Ji Z, Gonzalez F (2003) Artificial immune system (AIS) research in the last five years. In: The 2003 congress on evolutionary computation, 2003. CEC '03., vol 1. pp 123–130
Jameii SM, Kashani MH, Karimi R (2015) LASPEA: Learning automata-based strength pareto evolutionary algorithm for multi-objective optimization. Int J Comput Sci Telecommun 6(9):14–19
Yang X-S (2010) A new metaheuristic bat-inspired algorithm. In: Nature inspired cooperative strategies for optimization (NICSO 2010). Springer, pp 65–74
Yang X-S. Bat algorithm for multi-objective optimisation. arXiv e-prints, Accessed 01 Mar 2012. arXiv:1203.6571Y
Krishnanand KN, Ghose D (2009) Glowworm swarm optimization for simultaneous capture of multiple local optima of multimodal functions. Swarm Intell 3(2):87–124
Sarvizadeh R, Kashani MH, Zakeri FS, Jameii SM (2012) A novel bee colony approach to distributed systems scheduling. Int J Comput Appl 42(10):1–6
Saneja B, Rani R (2018) A hybrid approach for outlier detection in weather sensor data. In: 2018 IEEE 8th international advance computing conference (IACC). pp 321–326
Al-Madi N, Aljarah I, Ludwig S (2014) Parallel Glowworm Swarm Optimization Clustering Algorithm based on MapReduce
El-Alfy E-SM, Alshammari MA (2016) Towards scalable rough set based attribute subset selection for intrusion detection using parallel genetic algorithm in MapReduce. Simul Model Pract Theory 64(13):18–29
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
On behalf of all authors, the corresponding author states that there is no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Fathi, M., Haghi Kashani, M., Jameii, S.M. et al. Big Data Analytics in Weather Forecasting: A Systematic Review. Arch Computat Methods Eng 29, 1247–1275 (2022). https://doi.org/10.1007/s11831-021-09616-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11831-021-09616-4