Abstract
Activation function unit is an extremely important part of convolution neural networks; it is the nonlinear transformation that we do over the input data. Using hidden layer incorporating with a well-chosen activation function improves both the accuracy and the CNN convergence speed. This paper proposes a parametric rectified nonlinear function unit (PRenu). The proposed activation function is nearly similar to Relu. It returns \(x-\alpha \log (x+1)\) for positive values (\(\alpha \) is between 0 and 1) and zero for negative parts. In contrast to Relu that returns the same received gradient for all positive values in its back-propagation, the PRenu multiplies it by values between \(1-\alpha \) and 1 depending on the value with which each neuron was involved. The PRenu has been tested on three datasets: CIFAR-10, CIFAR-100 and Oxflower17, and compared to the activation function Relu. The experimental results show that using the proposed activation function PRenu, the CNN convergence is faster and the accuracy is also improved.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Arcos-García, Á., Álvarez García, J.A., Soria-Morillo, L.M.: Deep neural network for traffic sign recognition systems: an analysis of spatial transformers and stochastic optimisation methods. Neural Netw. 99, 158–165 (2018). https://doi.org/10.1016/j.neunet.2018.01.005
Chen, B., Jung, C.: Patch-based stereo matching using 3d convolutional neural networks. In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 3633–3637 (2018). https://doi.org/10.1109/ICIP.2018.8451527
Chen, Z., Ho, P.H.: Global-connected network with generalized relu activation. Pattern Recogn. 96(106), 961 (2019). https://doi.org/10.1016/j.patcog.2019.07.006
Clevert, D.A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (elus). 1511.07289 (2015)
Dong, X., Shen, J., Wang, W., Liu, Y., Shao, L., Porikli, F.: Hyperparameter optimization for tracking with continuous deep q-learning. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 518–527 (2018)
Dong, X., Shen, J., Wang, W., Shao, L., Ling, H., Porikli, F.: Dynamical hyperparameter optimization via deep reinforcement learning in tracking. IEEE Trans. Pattern Anal. Mach. Intell., pp 1–1 (2019)
Dong, X., Shen, J., Wu, D., Guo, K., Jin, X., Porikli, F.: Quadruplet network with one-shot learning for fast visual object tracking. IEEE Trans. Image Process. 28(7), 3516–3527 (2019)
Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: Gordon, G., Dunson, D., Dudík, M. (eds) Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, PMLR, Fort Lauderdale, FL, USA, Proceedings of Machine Learning Research, vol 15, pp 315–323 (2011)
He, K., Zhang, X., Ren, S., Sun, J. (2015). Delving deep into rectifiers: surpassing human-level performance on imagenet classification, 1502.01852
Jaafari, I.E., Ansari, M.E., Koutti, L., Mazoul, A., Ellahyani, A.: Fast spatio-temporal stereo matching for advanced driver assistance systems. Neurocomputing 194, 24–33 (2016). https://doi.org/10.1016/j.neucom.2016.02.010
Jaafari, I.E., Ansari, M.E., Koutti, L.: Fast edge-based stereo matching approach for road applications. Signal Image Video Process. 11, 267–274 (2017)
Maas, A. L.: Rectifier nonlinearities improve neural network acoustic models (2013)
Nair, V., Hinton, G. E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on International Conference on Machine Learning, Omnipress, USA, ICML’10, pp 807–814 (2010)
Nilsback, M.E., Zisserman, A.: A visual vocabulary for flower classification. IEEE Conf. Comput. Vis. Pattern Recogn. 2, 1447–1454 (2006)
Shen, J., Tang, X., Dong, X., Shao, L. (2019) Visual object tracking by hierarchical attention Siamese network. IEEE Trans. Cybern., pp 1–13
Shustanov, A., Yakimov, P.: Cnn design for real-time traffic sign recognition. Proc. Eng. 201, 718–725 (2017). https://doi.org/10.1016/j.proeng.2017.09.594
Soon, F.C., Khaw, H.Y., Chuah, J.H., Kanesan, J.: Vehicle logo recognition using whitening transformation and deep learning. Signal Image Video Process. 13, 111–119 (2019)
Wang, Z., Zhu, S., Li, Y., Cui, Z.: Convolutional neural network based deep conditional random fields for stereo matching. J. Vis. Commun. Image Represent. 40, 739–750 (2016). https://doi.org/10.1016/j.jvcir.2016.08.022
Xu, B., Wang, N., Chen, T., Li, M.: Empirical evaluation of rectified activations in convolutional network. 1505.00853 (2015)
Yin, W., Kann, K., Yu, M., Schütze, H.: Comparative study of cnn and rnn for natural language processing. 1702.01923 (2017)
Žbontar, J., LeCun, Y.: Stereo matching by training a convolutional neural network to compare image patches. J. Mach. Learn. Res. 17(65), 1–32 (2016)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
El Jaafari, I., Ellahyani, A. & Charfi, S. Parametric rectified nonlinear unit (PRenu) for convolution neural networks . SIViP 15, 241–246 (2021). https://doi.org/10.1007/s11760-020-01746-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11760-020-01746-9