Abstract
The Internet of Vehicles (IoV) is a communication paradigm that connects the vehicles to the Internet for transferring information between the networks. One of the key challenges in IoV is the management of a massive amount of traffic generated from a large number of connected IoT-based vehicles. Network clustering strategies have been proposed to solve the challenges of traffic management in IoV networks. Traditional optimization approaches have been proposed to manage the resources of the network efficiently. However, the nature of next-generation IoV environment is highly dynamic, and the existing optimization technique cannot precisely formulate the dynamic characteristic of IoV networks. Reinforcement learning is a model-free technique where an agent learns from its environment for learning the optimal policies. We propose an experience-driven approach based on an Actor-Critic based Deep Reinforcement learning framework (AC-DRL) for efficiently selecting the cluster head (CH) for managing the resources of the network considering the noisy nature of IoV environment. The agent in the proposed AC-DRL can efficiently approximate and learn the state-action value function of the actor and action function of the critic for selecting the CH considering the dynamic condition of the network.The experimental results show an improvement of 28% and 15% respectively, in terms of satisfying the SLA requirement and 35% and 14% improvement in throughput compared to the static and DQN approaches.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Availability of data and material
N/A
References
Aadil, F., Bajwa, K. B., Khan, S., Chaudary, N. M., & Akram, A. (2016). CACONET: Ant colony optimization (ACO) based clustering algorithm for VANET. PloS One, 11(5), e0154080.
Alouache, L., Nguyen, N., Aliouat, M., & Chelouah, R. (2019). Survey on IoV routing protocols: Security and network architecture. International Journal of Communication Systems, 32(2), e3849.
Chen, G., Li, C., Ye, M., & Jie, W. (2009). An unequal cluster-based routing protocol in wireless sensor networks. Wireless Networks, 15(2), 193–207.
Chen, H., Zhao, T., Li, C., & Guo, Y. (2019). Green Internet of vehicles: Architecture, enabling technologies, and applications. IEEE Access, 7, 179185–179198.
Contreras-Castillo, J., Zeadally, S., & Guerrero-Ibaez, J. A. (2017). Internet of vehicles: Architecture, protocols, and security. IEEE Internet of Things Journal, 5(5), 3701–3709.
Dai, Y., Du, X., Maharjan, S., Qiao, G., & Zhang, Y. (2019). Artificial intelligence empowered edge computing and caching for internet of vehicles. IEEE Wireless Communications, 26(3), 12–18.
Dutta, A. K., Elhoseny, M., Dahiya, V., & Shankar, K. (2020). An efficient hierarchical clustering protocol for multihop Internet of vehicles communication. Transactions on Emerging Telecommunications Technologies, 31(5), e3690.
Ebadinezhad, S., Dereboylu, Z., & Ever, E. (2019). Clustering-based modified ant colony optimizer for internet of vehicles (CACOIOV). Sustainability, 11(9), 2624.
El Khediri, S., Thaljaoui, A., Dallali, A., Fakhet, W., & Kachouri, A. (2018). An optimal clustering mechanism based on K-means for wireless sensor networks. In 2018 15th international multi-conference on systems, signals devices (SSD) (pp. 677–682). IEEE.
Elhoseny, M., Farouk, A., Zhou, N., Wang, M.-M., Abdalla, S., & Batle, J. (2017). Dynamic multi-hop clustering in a wireless sensor network: Performance improvement. Wireless Personal Communications, 95(4), 3733–3753.
Farhan, A., Ahsan, W., Rehman, Z. U., Shah, P. A., Rho, S., & Mehmood, I. (2018). Clustering algorithm for internet of vehicles (IoV) based on dragonfly optimizer (CAVDO). The Journal of Supercomputing, 74(9), 4542–4567.
Garbiso, J., Diaconescu, A., Coupechoux, M., & Leroy, B. (2016). Dynamic cluster size optimization in hybrid cellular-vehicular networks. In 2016 IEEE 19th international conference on intelligent transportation systems (ITSC) (pp. 557–563). IEEE.
Kaiwartya, O., Abdullah, A. H., Cao, Y., Altameem, A., Prasad, M., Lin, C.-T., et al. (2016). Internet of vehicles: Motivation, layered architecture, network model, challenges, and future aspects. IEEE Access, 4, 5356–5373.
Khan, M. F., Aadil, F., Maqsood, M., Bukhari, S. H. R., Hussain, M., & Nam, Y. (2018). Moth flame clustering algorithm for internet of vehicle (MFCA-IoV). IEEE Access, 7, 11613–11629.
Kuhnle, A., Kaiser, J.-P., Thei, F., Stricker, N., & Lanza, G. (2020). Designing an adaptive production control system using reinforcement learning. Journal of Intelligent Manufacturing, 7, 1–22.
Laroiya, N., & Lekhi, S. (2017). Energy efficient routing protocols in vanets. Advances in Computational Sciences and Technology, 10(5), 1371–1390.
Liu, Q., Cheng, L., Ozcelebi, T., Murphy, J., & Lukkien, J. (2019). Deep reinforcement learning for IoT network dynamic clustering in edge computing. Science, 10, 600–603.
Liu, K., Xu, X., Chen, M., Liu, B., Wu, L., & Lee, V. C. S. (2019). A hierarchical architecture for the future internet of vehicles. IEEE Communications Magazine, 57(7), 41–47.
Mao, H., Alizadeh, M., Menache, I., & Kandula, S. (2016). Resource management with deep reinforcement learning. In Proceedings of the 15th ACM workshop on hot topics in networks (pp. 50–56).
Mehboob, U., Qadir, J., Ali, S., & Vasilakos, A. (2016). Genetic algorithms in wireless networking: Techniques, applications, and issues. Soft Computing, 20(6), 2467–2501.
Naeem, F., Srivastava, G., & Tariq, M. (2020). A software defined network based fuzzy normalized neural adaptive multipath congestion control for Internet of Things. In IEEE transactions on network science and engineering.
Ning, Z., Huang, J., Wang, X., Rodrigues, J. J. P. C., & Guo, L. (2019). Mobile edge computing-enabled Internet of vehicles: Toward energy-efficient scheduling. IEEE Network, 33(5), 198–205.
Osamy, W., Salim, A., & Khedr, A. M. (2018). An information entropy based-clustering algorithm for heterogeneous wireless sensor networks. Wireless Networks, 6, 1–18.
Patel, N. J., & Jhaveri, R. H. (2015). Trust based approaches for secure routing in VANET: A Survey. Procedia Computer Science, 45, 592–601.
Qian, Y., Wu, J., Wang, R., Zhu, F., & Zhang, W. (2019). Survey on reinforcement learning applications in communication networks. Science, 4, 30–39.
Senouci, O., Harous, S., & Aliouat, Z. (2018). An efficient weight-based clustering algorithm using mobility report for IoV. In 2018 9th IEEE Annual ubiquitous computing, electronics mobile communication conference (UEMCON) (pp. 614–620). IEEE.
Shijie, W., & Yingfeng, Z. (2020). A credit-based dynamical evaluation method for the smart configuration of manufacturing services under Industrial Internet of Things. Journal of Intelligent Manufacturing, 51, 1–25.
Shirmohamadi, M., & Moradkhani, M. (2019). Reducing of energy Consuming in the Wireless sensor nets using clustering Protocol based on auto Organizing energies. No. 1532. EasyChair.
Sun, C., Zheng, S., Ma, Y., Chu, D., Yang, J., Zhou, Y., et al. (2020). An active safety control method of collision avoidance for intelligent connected vehicle based on driving risk perception. Journal of Intelligent Manufacturing, 89, 1–21.
Tarek, G., Abdelwahab, S., Elhoseny, M., & Hassanien, A. E. (2018). Trust-based secure clustering in WSN-based intelligent transportation systems. Computer Networks, 146, 151–158.
Wang, S., Zhao, Y., Jinlinag, X., Yuan, J., & Hsu, C.-H. (2019). Edge server placement in mobile edge computing. Journal of Parallel and Distributed Computing, 127, 160–168.
Yang, H., Alphones, A., Zhong, W.-D., Chen, C., & Xie, X. (2019). Learning-based energy-efficient resource management by heterogeneous RF/VLC for ultra-reliable low-latency industrial IoT networks. IEEE Transactions on Industrial Informatics, 16(8), 5565–5579.
Yang, H., Xie, X., & Kadoch, M. (2019). Intelligent resource management based on reinforcement learning for ultra-reliable and low-latency IoV Communication Networks. IEEE Transactions on Vehicular Technology, 68(5), 4157–4169.
Yousefi, S., Mousavi, M. S., & Fathy, M. (2006). Vehicular ad hoc networks (VANETs): Challenges and perspectives. In 2006 6th international conference on ITS telecommunications (pp. 761–766). IEEE.
Zhao, C., Dong, M., Ota, K., Li, J., & Jun, W. (2019). Edge-MapReduce-based intelligent information-centric IoV: Cognitive route planning. IEEE Access, 7, 50549–50560.
Funding
N/A
Author information
Authors and Affiliations
Contributions
Authors Abida Sharif and P.Li design this methodology and responsible for first draft. Authors Asim Saleem and Gunasekaran Manogran help in improving the writing. Authors S.Kadry and A.basit gives technical support in the analysis. Author MA Khan support data analysis.
Corresponding author
Ethics declarations
Conflict of interest
On the behalf of corresponding author, it is stated that authors not any conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Sharif, A., Li, J.P., Saleem, M.A. et al. A dynamic clustering technique based on deep reinforcement learning for Internet of vehicles. J Intell Manuf 32, 757–768 (2021). https://doi.org/10.1007/s10845-020-01722-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10845-020-01722-7