Skip to main content
SpringerLink
Log in

A Statistical Data-Filtering Method Proposed for Short-Term Load Forecasting Models

  • Original Article
  • Published:
Journal of Electrical Engineering & Technology Aims and scope Submit manuscript

Abstract

Reliability assessment of the SCADA-system based load data is necessary for improving accuracy of short-term load forecasting (STLF) methods in a distribution network (DN). Specifically, the reliability evaluation of the load data is to properly eliminate noise/outliers caused by random power consumption behaviors or the sudden change in load demand from industrial and residential customers in the DN. Thus, this paper proposes a novel statistical data-filtering method, working at an input data pre-processing stage, which will evaluate the reliability of input load data by analyzing all possible data confidence levels in order to filter-out the noise/outliers for accuracy improvement of different short-term load forecasting models. The proposed statistical data-filtering method is also compared to other existing data-filtering methods (such as Kalman Filter, Density-Based Spatial Clustering of Applications with Noise (DBSCAN), Discrete Wavelet Transform (DWT) and Singular Spectrum Analysis (SSA)). Moreover, several case studies of short-term load forecasting for a typical 22 kV distribution network in Vietnam are conducted with an Artificial Neural Network (ANN) model, a Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) model, a combined model of Long Short-Term Memory Network and Convolutional Neural Network (LSTM-CNN), and a conventional Autoregressive Integrated Moving Average (ARIMA) model to validate the statistical data-filtering method proposed. The achieved results demonstrate which the STLF using ANN, LSTM-RNN, LSTM-CNN, and ARIMA models with the statistical data-filtering method can all outperform those with the existing data-filtering methods. Additionally, the numerical results also indicate that in case the SCADA-based load data is normally distributed, time-series forecasting models should be more preferred than neural network models; otherwise, when the SCADA-based load data contains multiple normally distributed sub-datasets, neural network-based prediction models are highly recommended.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Kong W, Dong ZY, Jia Y, Hill DJ, Xu Y, Zhang Y (2017) Short-term residential load forecasting based on LSTM recurrent neural network. IEEE Trans Smart Grid 10(1):841–851. https://doi.org/10.1109/TSG.2017.2753802

    Google Scholar 

  2. Hippert HS, Pedreira CE, Souza RC (2001) Neural networks for short-term load forecasting: a review and evaluation. IEEE Trans Power Syst 16(1):44–55. https://doi.org/10.1109/59.910780

    Google Scholar 

  3. Pandey AS, Singh D, Sinha SK (2010) Intelligent hybrid wavelet models for short-term load forecasting. IEEE Trans Power Syst 25(3):1266–1273. https://doi.org/10.1109/TPWRS.2010.2042471

    Google Scholar 

  4. Qiang S, Pu Y (2019) Short-term power load forecasting based on support vector machine and particle swarm optimization. J Algorithm Comput Technol 13:1–8. https://doi.org/10.1177/1748301818797061

    MathSciNet  Google Scholar 

  5. Jujie W, Jianzhou W, Yaning Li, Suling Z, Jing Z (2014) Techniques of applying wavelet de-noising into a combined model for short-term load forecasting. Int J Electr Power Energy Syst 62:816–824. https://doi.org/10.1016/j.ijepes.2014.05.038

    Google Scholar 

  6. López MG, Valero S, Senabre C (2013) Short-term predictability of load series: characterization of load data Bases. IEEE Trans Power Syst 28(3):2466–2474. https://doi.org/10.1109/TPWRS.2013.2250317

    Google Scholar 

  7. Connor JT (1996) A robust neural network filter for electricity demand prediction. J Forecast 15(6):437–458

    Google Scholar 

  8. Melero JJ, Guerrero JJ, Beltran J, Pueyo C (2012) Efficient data filtering for wind energy assessment. IET Renew Power Gener 6(6):446–454. https://doi.org/10.1049/iet-rpg.2011.0288

    Google Scholar 

  9. Cao X, Dong S, Wu Z, Jing Y (2015) A data-driven hybrid optimization model for short-term residential load forecasting. In Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing (CIT/IUCC/DASC/PICOM), 2015 IEEE International Conference, pp: 283–287.

  10. Yun Z, Quan Z, Caixin S, Shaolan L, Yuming L, Yang S (2008) RBF neural network and ANFIS-based short-term load forecasting approach in real-time price environment. IEEE Trans Power Syst 23:853–858

    Google Scholar 

  11. Li H, Zhao Y, Zhang Z, Hu X (2015) Short-term load forecasting based on the grid method and the time series fuzzy load forecasting method. In International Conference on Renewable Power Generation (RPG 2015), pp: 1–6.

  12. Qingle P, Min Z (2010) Very short-term load forecasting based on neural network and rough set. In Intelligent Computation Technology and Automation (ICICTA), 2010 International Conference, pp: 1132–1135.

  13. Zhang R, Dong ZY, Xu Y, Meng K, Wong KP (2013) Short-term load forecasting of Australian National Electricity Market by an ensemble model of extreme learning machine. IET Gener Transm Distrib 7:391–397

    Google Scholar 

  14. Zhang R, Xu Y, Dong ZY Kong W (2016) A Composite k-Nearest Neighbor Model for Day-Ahead Load Forecasting with Limited Temperature Forecasts. In the IEEE General Meeting Boston.

  15. Al-Qahtani FH, Crone SF (2013) Multivariate k-nearest neighbour regression for time series data: a novel algorithm for forecasting UK electricity demand. In Neural Networks (IJCNN), The 2013 International Joint Conference on, pp: 1–8, 2013.

  16. Tian C, Ma J, Zhang C, Zhan PA (2018) Deep neural network model for short-term load forecast based on long short-term memory network and convolutional neural network. Energies 11(12):3493

    Google Scholar 

  17. Ryu S, Noh J, Kim H (2016) Deep neural network based demand side short term load forecasting. In The 2016 IEEE International Conference on Smart Grid Communications (SmartGridComm), pp: 308–313.

  18. Mocanu E, Nguyen PH, Gibescu M, Kling WL (2016) Deep learning for estimating building energy consumption. Sustain Energy Grids Netw 6:91–99

    Google Scholar 

  19. Gastaldi M, Lamedica R, Nardecchia A, Prudenzi A (2004) Short-term forecasting of municipal load through a Kalman Filtering based approach. IEEE PES Power Syst Conf Expos 3:1453–1458. https://doi.org/10.1109/PSCE.2004.1397538

    Google Scholar 

  20. Al-Hamadi HM, Soliman SA (2006) Fuzzy short-term electric load forecasting using Kalman filter. IEE Proc Gener Transm Distrib 153(2):217–227. https://doi.org/10.1049/ip-gtd:20050088

    Google Scholar 

  21. Guan C, Luh PB, Michel LD, Chi Z (2013) Hybrid Kalman Filters for very short-term load forecasting and prediction interval estimation. IEEE Trans Power Syst 28(4):3806–3817. https://doi.org/10.1109/TPWRS.2013.2264488

    Google Scholar 

  22. Ghofrani M, Hassanzadeh M, Etezadi-Amoli M, Fadali MS (2011) Smart meter based short-term load forecasting for residential customers. N Am Power Symp. https://doi.org/10.1109/NAPS.2011.6025124

    Google Scholar 

  23. Tai N, Jürgen S, Wu H (2006) Techniques of applying wavelet transform into combined model for short-term load forecasting. Electric Power Syst Res 76(6–7):525–533. https://doi.org/10.1016/j.epsr.2005.07.003

    Google Scholar 

  24. Connor JT (1996) A robust neural network filter for electricity demand prediction. J Forecast 15:437–458

    Google Scholar 

  25. Ko CN, Lee CM (2013) Short-term load forecasting using SVR (support vector regression)-based radial basis function neural network with dual extended Kalman filter. Energy 49:413–422

    Google Scholar 

  26. Ester M, Kriegel HP, Sander J, Xu X (1996) A density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, pp: 226–231.

  27. Amr M, Noha AY, Nagwa MEM (2011) Discovering Clusters with Arbitrary Shapes and Densities in Data Streams. In: The 10th International Conference on Machine Learning and Applications and Workshops.

  28. Zheng H, Wang Z, Zhang L, Wang Q (2010) Clustering algorithm based on characteristics of density distribution. In the 2nd International Conference on Advanced Computer Control.

  29. Mohammad FH, Mohamed H, Abdalla S (2015) DDBSCAN: Different Densities-Based Spatial Clustering of Applications with Noise. In the International Conference on Control, Instrumentation, Communication and Computational Technologies (ICCICCT).

  30. Zhen W, Rui Z, Jianzhong Q, Bo Y (2019) DBSVEC: Density-based clustering using support vector expansion. In the IEEE 35th International Conference on Data Engineering (ICDE).

  31. Huang CM, Huang YC (2002) Combined wavelet-based net- works and game-theoretical decision approach for real-time power dis- patch. IEEE Trans Power Syst 17(8):633–639

    Google Scholar 

  32. Oonsivilai A, El-Hawary ME (1999) Wavelet neural network based short term load forecasting of electric power system commercial load. In Proceedings of IEEE Canadian Conference on Electrical and Computer Enginerring, Edmonton, Canada, pp: 1223–1228.

  33. Kim C-I, Yu I-K, Song YH (2002) Kohonen neural networks and wavelet transform based approach to short-term load forecasting. Elect Electr Power Syst Res 63(3):169–176

    Google Scholar 

  34. Zhang B-L, Dong Z-Y (2001) An adaptive neural-wavelet model for short term load forecasting. Electr Power Syst Res 59(2):121–129

    Google Scholar 

  35. Zhang B, Coggins R, Jabri MA, Dersch D (2001) Multiresolution fore- casting for futures trading using wavelet decompositions. IEEE Trans. Neural Network pp: 765–775.

  36. Guan C et al (2013) Very short-term load forecasting: wavelet neural networks with data pre-filtering. IEEE Trans Power Syst 28(1):30–41

    Google Scholar 

  37. Abinet TE, Jianhua Z, Dehua Z (2018) Short-term photovoltaic solar power forecasting using a hybrid Wavelet-PSO-SVM model based on SCADA and Meteorological information. Renew Energy 118:357–367

    Google Scholar 

  38. Bashir ZA, El-Hawary ME (2009) Applying wavelets to short-term load forecasting using PSO-based neural networks. IEEE Trans Power Systems 24(1):20–27

    Google Scholar 

  39. Li S, Wang P, Goel L (2016) A novel wavelet-based ensemble method for short-term load forecasting with hybrid neural networks and feature selection. IEEE Trans Power Syst 31(3):1788–1798

    Google Scholar 

  40. Ajay SP, Devender S, Sunil KS (2010) Intelligent hybrid wavelet models for short-term load forecasting. IEEE Trans Power Syst 25(3):1266–1273

    Google Scholar 

  41. Wang J et al (2014) Techniques of applying wavelet de-noising into a combined model for short-term load forecasting. Electr Power Energy Syst 62:816–824

    Google Scholar 

  42. Chen Y et al (2010) Short-term load forecasting: similar day-based wavelet neural networks. IEEE Trans Power Syst 25(1):322–330

    Google Scholar 

  43. Bowen L, Jing Z, Yu H, Yang W (2017) Short-term load-forecasting method based on wavelet decomposition with second-order gray neural network model combined with ADF test. IEEE Access 5:16324–16331

    Google Scholar 

  44. Li S, Wang P, Goel L (2015) Short-term load forecasting by wavelet transform and evolutionary extreme learning machine. Electric Power Syst Res 122:96–103

    Google Scholar 

  45. Chaturvedi DK, Sinha AP, Malik OP (2015) Short term load forecast using fuzzy logic and wavelet transform integrated generalized neural network. Electr Power Energy Syst 67:230–237

    Google Scholar 

  46. Xiaobo Z, Jianzhou W, Kequan Z (2017) Short-term electric load forecasting based on singular spectrum analysis and support vector machine optimized by Cuckoo search algorithm. Electric Power Syst Res 146:270–285. https://doi.org/10.1016/j.epsr.2017.01.035

    Google Scholar 

  47. Liu L et al (2018) Short-term local prediction of wind speed and wind power based on singular spectrum analysis and locality-sensitive hashing. J Mod Power Syst Clean Energy 6(2):317–329

    Google Scholar 

  48. Miranian A, Abdollahzade M, Hassani H (2013) Day-ahead electricity price analysis and forecasting by singular spectrum analysis. IET Gener Transm Distrib 7(4):337–346

    Google Scholar 

  49. Harish V, Sujil A, Rajesh K (2018) A singular spectrum analysis based approach to price forecasting for a day ahead electricity market. In The 8th IEEE India International Conference on Power Electronics (IICPE).

  50. Hossein AV et al (2007) Combination of singular spectrum analysis and autoregressive model for short term load forecasting. IEEE Lausanne Power Tech. https://doi.org/10.1109/PCT.2007.4538467

    Google Scholar 

  51. Shruti S, et al. (2014) Data analysis using principal component analysis. In the 2014 International Conference on Medical Imaging, m-Health and Emerging Communication Systems (MedCom).

  52. Haykin S (1999) Neural networks: a comprehensive foundation. Prentice-Hall, New Jersey

    MATH  Google Scholar 

  53. Iason-Ioannis Ch (2018) Very short-term probabilistic demand forecasting at high aggregation level for the mitigation of balancing costs,” Master thesis, ETH Zürich.

  54. Alexander E, Marcus V, Sahin A (2019) Short-term probabilistic load forecasting at low aggregation levels using convolutional neural networks. IEEE Milan PowerTech. https://doi.org/10.1109/PTC.2019.8810811

    Google Scholar 

  55. Hagan MT, Menhaj MB (1994) Training feedforward networks with the Marquardt algorithm. IEEE Trans Neural Netw 5(6):989–993

    Google Scholar 

  56. Catalão JPS, Pousinho HMI, Mendes VMF (2011) Short-term wind power forecasting in Portugal by neural networks and wavelet transform. Renew Energy 36:1245–1251

    Google Scholar 

  57. Khashei M, Bijari MA (2011) Novel hybridization of artificial neural networks and ARIMA models for time series forecasting. Appl Soft Comput 11:2664–2675

    Google Scholar 

  58. Box GE, Jenkins GM (1976) Time series analysis: forecasting and control. Revised Edition, Holden Day, San Francisco, ISBN 9780816211043

  59. Rosca RE (2011) Stationary and non-stationary time series. USV Ann Econ Public Adm 10:177–186

    Google Scholar 

  60. Yuan C, Liu S, Fang Z (2016) Comparison of China's primary energy consumption forecasting by using ARIMA (the autoregressive integrated moving average) model and GM (1, 1) model. Energy 100:384–390

    Google Scholar 

  61. Bengio Y, Courville A, Vincent P (2013) Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell 35:1798–1828

    Google Scholar 

  62. Bengio Y, Simard P, Frasconi P (1994) Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Networks 5:157–166

    Google Scholar 

  63. Du S, Li T, Gong X, Yang Y, Horng SJ (2017) Traffic flow forecasting based on hybrid deep learning framework. In Proceedings of the 12th International Conference on Intelligent Systems and Knowledge Engineering, Nanjing, China, pp: 1–6.

  64. Zeiler MD, Ranzato M, Monga R (2013) On rectified linear units for speech processing. In: The 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) pp: 3517–3521.

Download references

Acknowledgement

The authors gratefully acknowledge financial support from Mr. Phuc Duy Le and Mr. Tien Minh Cao for accomplishing this project of Ho Chi Minh City University of Technology (HUTECH)-Vietnam. The authors would also like to send many sincere thanks to Dr. Hung Nguyen and Mr. Duy Anh Pham for their valuable contributions to revise this paper.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering (ECE), Faculty of Engineering, Vietnamese-German University (VGU), Thu Dau Mot City, Binh Duong Province, 75100, Vietnam

    Duong Minh Bui

  2. Department of Industrial and Systems Engineering, Chung Yuan Christian University (CYCU), Taoyuan City, Taiwan

    Trang Thi Pham

  3. Department of Computer Science, Bonn-Rhein-Sieg University of Applied Sciences, North Rhine-Westphalia, Germany

    Duy Anh Pham

  4. Institute of Engineering, Ho Chi Minh City University of Technology (HUTECH), Ho Chi Minh City, 70000, Vietnam

    Phuc Duy Le, Tien Minh Cao & Hung Nguyen

Authors
  1. Duong Minh Bui
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Phuc Duy Le
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tien Minh Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hung Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Trang Thi Pham
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Duy Anh Pham
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Duong Minh Bui.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bui, D.M., Le, P.D., Cao, T.M. et al. A Statistical Data-Filtering Method Proposed for Short-Term Load Forecasting Models. J. Electr. Eng. Technol. 15, 1947–1967 (2020). https://doi.org/10.1007/s42835-020-00460-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42835-020-00460-3

Keywords

Search

Navigation