Abstract
Ocean sensor data prediction has become a promising means for smart ocean monitoring. In alternative solutions, deep neural networks (DNNs) are considered as a good choice. The determination of activation functions in DNNs has a significant effect on training speed and nonlinear approximation. In this paper, the effect of activation functions on a deep computing model called deep belief echo-state network (DBEN) is studied in the scenario of ocean time series prediction. Here, different forms, including hyperbolic tangent, rectified linear unit, exponential linear unit, swish, softplus and their variants, are considered. The purpose is to investigate, from the perspectives of accuracy and training efficiency, whether certain activation function in DBEN is completely universal for the different tasks of ocean sensor data processing or not. On a great deal of real-world ocean time series of different characteristics, the results show that the selection of activation functions in DBEN is task-related. Specially, these newly introduced activation functions are more beneficial to the accurate predictions for conventional and chemical data sets compared with sigmoid benchmark. The statistical analysis further verifies this finding.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Waterston, J.; Rhea, J.; Peterson, S.; Bolick, L.; Ayers, J.; Ellen, J.: Ocean of Things: affordable maritime sensors with scalable analysis. In: OCEANS, pp. 1–6. IEEE, Marseille, France, (2019)
Yang, J.; Wen, J.; Wang, Y.; Jiang, B.; Wang, H.; Song, H.: Fog-based marine environmental information monitoring towards Ocean of Things. IEEE Internet Things J. 7(5), 4238–4247 (2019)
Trasvina-Moreno, C.A.; Blasco, R.; Marco, A.; Casas, R.; Trasvina-Castro, A.: Unmanned aerial vehicle based wireless sensor network for marine-coastal environment monitoring. Sensors 17(3), 460 (2017)
Sun, X.; Gui, G.; Li, Y.; Liu, R.P.; An, Y.: ResInNet: a novel deep neural network with feature reuse for Internet of things. IEEE Internet Things J. 6(1), 679–691 (2019)
Liu, J., et al.: Automatic and accurate prediction of key water quality parameters based on SRU deep learning in mariculture. In: 2018 IEEE International Conference on Advanced Manufacturing (ICAM), Yunlin, pp. 437–440 (2018)
Hu, Z.; Zhang, Y.; Zhao, Y.; Xie, M.; Zhong, J.; Tu, Z.; Liu, J.: A water quality prediction method based on the deep LSTM network considering correlation in smart mariculture. Sensors 19(6), 1420–1439 (2019)
Qin, M.; Li, Z.; Du, Z.: Red tide time series forecasting by combining ARIMA and deep belief network. Knowl.-Based Syst. 125, 39–52 (2017)
Li, Z.; et al.: Smoothed deep neural networks for marine sensor data prediction. IEEE Access 8, 22802–22811 (2020)
Abualigah, L.M.Q.: Feature Selection and Enhanced Krill Herd Algorithm for Text Document Clustering. Springer, Berlin (2019)
Abualigah, L.M.; Khader, A.T.: Unsupervised text feature selection technique based on hybrid particle swarm optimization algorithm with genetic operators for the text clustering. J. Supercomput. 73(11), 4773–4795 (2017)
Lee, J.; Shridhar, K.; Hayashi, H.; et al.: ProbAct: a probabilistic activation function for deep neural networks, arXiv preprint arXiv:1905.10761 (2019)
He, K.; Zhang, X.; Ren, S.; Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(9), 1904–1916 (2015)
Anthimopoulos, M.; Christodoulidis, S.; Ebner, L.; et al.: Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans. Med. Imaging 25(5), 1207–1216 (2016)
Zhang, Y.D.; Pan, C.; Sun, J.; et al.: Multiple sclerosis identification by convolutional neural network with dropout and parametric ReLU. J. Comput. Sci. 28(2018), 1–10 (2018)
Nwankpa, C.; Ijomah, W.; Gachagan, A.; et al.: Activation functions: comparison of trends in practice and research for deep learning, arXiv preprint arXiv:1811.03378 (2018)
Sun, X.; Li, T.; Li, Q.; Huang, Y.; Li, Y.: Deep belief echo-state network and its application to time series prediction. Knowl.-Based Syst. 130, 17–29 (2017)
Chen, Z.; Li, W.: Multisensor feature fusion for bearing fault diagnosis using sparse autoencoder and deep belief network. IEEE Trans. Instrum. Meas. 66(7), 1693–1702 (2017)
Rahman, N.H.A.; Zobaa, A.F.: Integrated mutation strategy with modified binary PSO algorithm for optimal PMUs placement. IEEE Trans. Ind. Inf 13(6), 3124–3133 (2017)
Xiao, L.; Zhang, Y.; Hu, Z.; Dai, J.: Performance benefits of robust nonlinear zeroing neural network for finding accurate solution of Lyapunov equation in presence of various noises. IEEE Trans. Ind. Inf. 15(9), 5161–5171 (2019)
Gulcehre, C.; Moczulski, M.; Denil, M.; Bengio, Y.: Noisy activation functions. In: International Conference on Machine Learning (2016)
Canals, V.; et al.: A new stochastic computing methodology for efficient neural network implementation. IEEE Trans. Neural Netw. Learn. Syst. 27(3), 551–564 (2015)
Liew, S.S.; Khalil-Hani, M.; Bakhteri, R.: Bounded activation functions for enhanced training stability of deep neural networks on visual pattern recognition problems. Neurocomputing 216, 718–734 (2016)
Dittmer, S.; Emily, J.; Maass, P.: Singular values for ReLU layers. IEEE Trans. Neural Netw. Learn. Syst. (2019). https://doi.org/10.1109/TNNLS.2019.2945113
Gagana, B.; et al.: Activation function optimizations for capsule networks. In: 2018 International Conference on Advances in Computing, Communications and Informatics (ICACCI), pp. 1172–1178. IEEE (2018)
Ying, Y.; Su, J.; Shan, P.; Miao, L.; Wang, X.; Peng, S.: Rectified exponential units for convolutional neural networks. IEEE Access 7, 101633–101640 (2019)
Yang, H.; Jang, H.; Kim, T.; Lee, B.: Non-temporal lightweight fire detection network for intelligent surveillance systems. IEEE Access 7, 169257–169266 (2019)
Chen, Y.; Mai, Y.; Xiao, J.; Zhang, L.: Improving the antinoise ability of DNNs via a bio-inspired noise adaptive activation function rand softplus. Neural Comput. 31(6), 1215–1233 (2019)
Li, Z.; Wang, N.; Li, Y.; Sun, X.; Huo, M.; Zhang, H.: Collective efficacy of support vector regression with smoothness priority in marine sensor data prediction. IEEE Access 7, 10308–10317 (2019)
Aghelpour, P.; Varshavian, V.: Evaluation of stochastic and artificial intelligence models in modeling and predicting of river daily flow time series. Stoch. Environ. Res. Risk Assess. 34(1), 33–50 (2020)
Abualigah, L.: Multi-verse optimizer algorithm: a comprehensive survey of its results, variants, and applications. Neural Comput. Appl. 32, 12381–12401 (2020)
Funding
This work was supported in part by the Natural Science Foundation of Hebei Province under Grant F2018209181, in part by the S&T Major Project of the Science and Technology Ministry of China under Grant 2017YFE0135700, and in part by the Science and Technology Project of Tangshan (19150230E).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
CONFLICTS OF INTEREST The authors declare that they have no conflicts of interest.
Rights and permissions
About this article
Cite this article
Li, Z., Wang, J., Cao, D. et al. Investigating Neural Activation Effects on Deep Belief Echo-State Networks for Prediction Toward Smart Ocean Environment Monitoring. Arab J Sci Eng 46, 3913–3923 (2021). https://doi.org/10.1007/s13369-020-05319-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13369-020-05319-3