Skip to main content
SpringerLink
Log in

An Intelligent Auto-Organizing Aerial Robotic Sensor Network System for Urban Surveillance

  • Regular Paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

We have developed and demonstrated an intelligent auto-organizing aerial robotic sensor network system composed of cameras installed on UAVs and ground fixtures for urban surveillance using the decentralised control paradigm. The system can auto-organize to meet user requirements, changes to user requirements and when new UAVs or ground cameras are added or are lost due to failure. User designated targets can be automatically tracked and handed over from one camera to another, to ensure continuous tracking. The cameras can also self adjust their positions and pointing angles to cooperatively cover areas of interest completely. We have achieved this by designing a software architecture for managing multiple interacting algorithms online and in parallel, facilitating incremental software development and debugging. We have adapted algorithms to become computationally efficient and effective for real world use online and on-board the UAVs, and ensured that the integration of video analytics and tactical behaviours work in real world settings. We have also achieved near optimal, online close coordination of the UAVs and ground cameras. To the best of our knowledge, this is the first proof-of-concept of a fully integrated intelligent auto-organizing aerial robotic sensor network system of cameras installed on UAVs and ground fixtures for urban surveillance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Availability of Data and Materials

The authors will be glad to furnish supplementary materials consisting of videos of simulation runs, flight tests of individual capabilities, and integrated demonstration scenarios upon request by email.

References

  1. Chung, S.-J., Paranjape, A.A., Dames, P., Shen, S., Kumar, V.: A survey on aerial swarm robotics. IEEE Trans. Robot. 34(4), 837–855 (2018)

    Article  Google Scholar 

  2. Zhu, B., Xie, L., Han, D., Meng, X., Teo, R.: A survey on recent progress in control of swarm systems. Sci. China Inform. Sci. 60(7), 070201 (2017)

    Article  MathSciNet  Google Scholar 

  3. Brunet, L., Choi, H.-L., How, J.: Consensus-based auction approaches for decentralized task assignment. In: AIAA guidance, navigation and control conference and exhibit, p 6839 (2008)

  4. Choi, H.-L., Brunet, L., How, J.P.: Consensus-based decentralized auctions for robust task allocation. IEEE Trans Robot 25(4), 912–926 (2009)

    Article  Google Scholar 

  5. Johnson, L., Ponda, S., Choi, H.-L., How, J.: Improving the efficiency of a decentralized tasking algorithm for uav teams with asynchronous communications. In: AIAA Guidance Navigation, and Control Conference, p 8421 (2010)

  6. Johnson, L., Ponda, S., Choi, H.-L., How, J.. In: Infotech@ Aerospace 2011, p 1441 (2011)

  7. Ponda, S.S., Johnson, L.B., Kopeikin, A.N., Choi, H.-L., How, J.P.: Distributed planning strategies to ensure network connectivity for dynamic heterogeneous teams. IEEE J. Select. Areas Commun. 30 (5), 861–869 (2012)

    Article  Google Scholar 

  8. Choi, H.-L., Whitten, A.K., How, J.P.: Decentralized task allocation for heterogeneous teams with cooperation constraints. In: Proceedings of the 2010 American Control Conference, pp 3057–3062. IEEE (2010)

  9. Whitten, A.K., Choi, H. -L., Johnson, L.B., How, J.P.: Decentralized task allocation with coupled constraints in complex missions. In: Proceedings of the 2011 American Control Conference, pp 1642–1649. IEEE (2011)

  10. Ponda, S.S., Johnson, L.B., How, J.P.: Distributed chance-constrained task allocation for autonomous multi-agent teams. In: 2012 American Control Conference (ACC), pp 4528–4533. IEEE (2012)

  11. Johnson, L., Choi, H.-L., Ponda, S., How, J.P.: Allowing non-submodular score functions in distributed task allocation. In: 2012 IEEE 51st IEEE Conference on Decision and Control (CDC), pp 4702–4708. IEEE (2012)

  12. Ponda, S.S., Johnson, L.B., How, J.P.: Risk allocation strategies for distributed chance-constrained task allocation. In: 2013 American Control Conference, pp 3230–3236. IEEE (2013)

  13. Buckman, N., Choi, H.-L., How, J.P.: Partial replanning for decentralized dynamic task allocation. In: AIAA Scitech 2019 Forum, p 0915 (2019)

  14. Bertuccelli, L., Choi, H.-L., Cho, P., How, J.: Real-time multi-uav task assignment in dynamic and uncertain environments. In: AIAA guidance, navigation, and control conference, p 5776 (2009)

  15. How, J.P., Fraser, C., Kulling, K.C., Bertuccelli, L.F., Toupet, O., Brunet, L., Bachrach, A., Roy, N.: Increasing autonomy of uavs. IEEE Robot. Autom. Magazine 16(2), 43–51 (2009)

    Article  Google Scholar 

  16. Ponda, S., Redding, J., Choi, H.-L., How, J.P., Vavrina, M., Vian, J.: Decentralized planning for complex missions with dynamic communication constraints. In: Proceedings of the 2010 American Control Conference, pp 3998–4003. IEEE (2010)

  17. Hunt, S., Meng, Q., Hinde, C.J.: An extension of the consensus-based bundle algorithm for multi-agent tasks with task based requirements. In: 2012 11th International Conference on Machine Learning and Applications, vol. 2, pp 451–456. IEEE (2012)

  18. Hunt, S., Meng, Q., Hinde, C.J.: An extension of the consensus-based bundle algorithm for group dependant tasks with equipment dependencies. In: International Conference on Neural Information Processing, pp 518–527. Springer (2012)

  19. Hunt, S., Meng, Q., Hinde, C., Huang, T.: A consensus-based grouping algorithm for multi-agent cooperative task allocation with complex requirements. Cognitive computation 6(3), 338–350 (2014)

    Article  Google Scholar 

  20. Huang, S., Teo, R.S.H., Leong, W.W.L., Martinel, N., Forest, G.L., Micheloni, C.: Coverage control of multiple unmanned aerial vehicles: A short review. Unmanned Syst. 6(2), 131–144 (2018)

    Article  Google Scholar 

  21. Schwager, M., Julian, B.J., Angermann, M., Rus, D.: Eyes in the sky: Decentralized control for the deployment of robotic camera networks. Proc. IEEE 99(9), 1541–1561 (2011)

    Article  Google Scholar 

  22. Huang, S., Teo, R.S.H., Leong, W.L.: Distributed coverage control for multiple unmanned multirotors with downward facing pan-tilt-zoom-cameras. In: International Conference on Unmanned Aircraft Systems. Conference Proceedings, pp 744–751. IEEE (2018)

  23. Huang, S., Yang, H., Leong, W.L., Teo, R.S.H.: Improved multi-camera coverage control of unmanned multirotors. In: International Conference on Unmanned Aircraft Systems, pp 1103–1112. IEEE (2020)

  24. Huang, S., Teo, R.S.H., Tan, K.K.: Collision avoidance of multi unmanned aerial vehicles: A review. Annu. Rev. Control. 48, 147–164 (2019)

    Article  MathSciNet  Google Scholar 

  25. Kuwata, Y., Wolf, M.T., Zarzhitsky, D., Huntsberger, T.L.: Safe maritime autonomous navigation with colregs, using velocity obstacles. IEEE J. Ocean. Eng. 39(1), 110–119 (2013)

    Article  Google Scholar 

  26. Rimon, E., Koditschek, D.E.: Exact robot navigation using artificial potential functions. Departmental Papers (ESE) 323 (1992)

  27. Mehdi, S.B., Cichella, V., Marinho, T., Hovakimyan, N.: Collision avoidance in multi-vehicle cooperative missions using speed adjustment. In: 2017 IEEE 56th Annual Conference on Decision and Control (CDC), pp 2152–2157. IEEE (2017)

  28. Mylvaganam, T., Sassano, M., Astolfi, A.: A differential game approach to multi-agent collision avoidance. IEEE Trans. Autom. Control 62(8), 4229–4235 (2017)

    Article  MathSciNet  Google Scholar 

  29. Morgan, D., Subramanian, G.P., Chung, S.-J., Hadaegh, F.Y.: Swarm assignment and trajectory optimization using variable-swarm, distributed auction assignment and sequential convex programming. Int. J. Robot. Res. 35(10), 1261–1285 (2016)

    Article  Google Scholar 

  30. Zammit, C., Van Kampen, E.-J.: Comparison of a* and rrt in real–time 3d path planning of uavs. In: AIAA Scitech 2020 Forum, p 0861 (2020)

  31. Huang, H.-P., Chung, S.-Y.: Dynamic visibility graph for path planning. In: International Conference on Intelligent Robots and Systems, vol. 3, pp 2813–2818. IEEE (2004)

  32. Huang, S., Teo, R.S.H.: Computationally efficient visibility graph-based generation of 3d shortest collision-free path among polyhedral obstacles for unmanned aerial vehicles. In: 2019 International Conference on Unmanned Aircraft Systems (ICUAS), pp 1218–1223. IEEE (2019)

  33. Zou, Z., Shi, Z., Guo, Y., Ye, J.: Object detection in 20 years: A Survey. ArXiv e-prints 1–39 (2019)

  34. Yao, R., Lin, G., Xia, S., Zhao, J., Zhou, Y.: Video Object Segmentation and Tracking: A Survey, ArXiv e-prints (2019)

  35. Karanam, S., Gou, M., Wu, Z., Rates-Borras, A., Camps, O., Radke, R.J.: A comprehensive evaluation and benchmark for person re-identification: features, Metrics, and Datasets. IEEE Trans. Pattern Anal. Mach. Intell. 1–14 (2018)

  36. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: SSD: Single Shot multibox detector, European Conferece on Computer Vision. In: LNCS, vol. 9905, pp 21–37 (2016)

  37. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: International Conference on Computer Vision and Pattern Recognition, pp 580–587 (2014)

  38. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-CNN: Towards Real-Time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017)

    Article  Google Scholar 

  39. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: International Conference on Computer Vision and Pattern Recognition (2016)

  40. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal loss for dense object detection. International Conference on Computer Vision 2017, 2999–3007 (2017)

    Google Scholar 

  41. Schmidhuber, J.: Deep Learning in neural networks: An overview. Neural Netw. 61, 85–117 (2015)

    Article  Google Scholar 

  42. Redmon, J., Farhadi, A.: YOLOV3: An incremental improvement. In: International Conference on Computer Vision and Pattern Recognition (2017)

  43. Wu, Y., Lim, J., Yang, M.-H.: Online Object Tracking: A Benchmark. In: CVPR, pp 2411–2418 (2013)

  44. Huang, L., Zhao, X., Huang, K.: Got-10k: A large high-diversity benchmark for generic object tracking in the wild. In: IEEE Transactions on Pattern Analysis and Machine Intelligence (2019)

  45. Kristan, M., Matas, J., Leonardis, A., Felsberg, M., Pflugfelder, R., Kamarainen, J.-K., Cehovin, L., Drbohlav, O., Lukezic, A., Berg, A.: A eldesokey, the seventh visual object tracking VOT2019 challenge results. In: International Conference on Computer Vision Workshops (2019)

  46. Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: High-speed tracking with kernelized correlation filters. IEEE Trans. Pattern Anal. Machine Intell. 37(3), 583–596 (2015)

    Article  Google Scholar 

  47. Danelljan, M., Bhat, G., Khan, F.S., Felsberg, M.: ECO: Efficient Convolution Operators For tracking. In: International Conference on Computer Vision and Pattern Recgnition (2017)

  48. Wang, Q., Gao, J., Xing, J., Zhang, M., Hu, W.: DCFNet: Discriminant correlation filters network for visual tracking, arXiv preprint (2017)

  49. Dunnhofer, M., Martinel, N., Foresti, G.L., Micheloni, C.: Visual tracking by means of deep reinforcement learning and an expert demonstrator. In: International Conference on Computer Vision Workshops (2019)

  50. Martinel, N., Micheloni, C.: Classification of Local Eigen-Dissimilarities for Person Re-Identification. IEEE Signal Process. Lett. 22(4), 455–459 (2015)

    Article  Google Scholar 

  51. Martinel, N.: Accelerated low-rank sparse metric learning for person re-identification. Pattern Recogn. Lett. 112, 234–240 (2018)

    Article  Google Scholar 

  52. Saini, K., Kansal, K., Venkata, A.S.: Airborne visual tracking and reidentification system. J. Electron. Imag. 28(2), 1–10 (2019)

    Article  Google Scholar 

  53. Maza, I., Caballero, F., Capitán, J., Martínez-de Dios, J.R., Ollero, A.: Experimental results in multi-uav coordination for disaster management and civil security applications. J. Intell. Robot. Syst. 61(1-4), 563–585 (2011)

    Article  Google Scholar 

  54. Smith, R.G.: The contract net protocol: High-level communication and control in a distributed problem solver. IEEE Trans. Comput. 12, 1104–1113 (1980)

    Article  Google Scholar 

  55. Saska, M., Chudoba, J., Přeučil, L., Thomas, J., Loianno, G., Třešňák, A., Vonásek, V., Kumar, V.: Autonomous deployment of swarms of micro-aerial vehicles in cooperative surveillance. In: 2014 International Conference on Unmanned Aircraft Systems (ICUAS), pp 584–595. IEEE (2014)

  56. Scherer, J., Yahyanejad, S., Hayat, S., Yanmaz, E., Andre, T., Khan, A., Vukadinovic, V., Bettstetter, C., Hellwagner, H., Rinner, B.: An autonomous multi-uav system for search and rescue. In: Proceedings of the First Workshop on Micro Aerial Vehicle Networks Systems, and Applications for Civilian Use, pp 33–38 (2015)

  57. Yanmaz, E., Yahyanejad, S., Rinner, B., Hellwagner, H., Bettstetter, C.: Drone networks: Communications, coordination, and sensing. Ad Hoc Netw. 68, 1–15 (2018)

    Article  Google Scholar 

  58. Gu, J., Su, T., Wang, Q., Du, X., Guizani, M.: Multiple moving targets surveillance based on a cooperative network for multi-uav. IEEE Commun. Mag. 56(4), 82–89 (2018)

    Article  Google Scholar 

  59. de Moraes, R.S., de Freitas, E.P.: Multi-uav based crowd monitoring system. IEEE Trans. Aerosp. Electron. Syst. 56(2), 1332–1345 (2019)

    Article  Google Scholar 

  60. Martinel, N., Foresti, G.L., Micheloni, C.: Deep Pyramidal Pooling with Attention for Person Re-Identification. In: IEEE Transaction on Image Processing (2020)

  61. Dunnhofer, M., Antico, M., Sasazawa, F., Takeda, Y., Camps, S., Martinel, N., Micheloni, C., Carneiro, G., Fontanarosa, D.: Siam-u-net: encoder-decoder siamese network for knee cartilage tracking in ultrasound images. Medical Image Analysis 60, 101631 (2020)

    Article  Google Scholar 

  62. Martinel, N., Dunnhofer, M., Foresti, G.L., Micheloni, C.: Person Re-identification via unsupervised transfer of learned visual representations. In: International Conference on Distributed Smart Cameras, pp 1–6 (2017)

  63. Martinel, N., Foresti, G.L., Micheloni, C.: Distributed person re-identification through network-wise rank fusion consensus. Pattern Recognit. Lett. 124, 63–73 (2019)

    Article  Google Scholar 

  64. Martinel, N., Foresti, G.L., Micheloni, C.: Aggregating deep pyramidal representations for person Re-Identification. In: International Conference on Computer Vision and Pattern Recognition (2019)

  65. Tan, C.Y., Huang, S., Tan, K.K., Teo, R.S.H., Liu, W.Q., Lin, F.: Collision avoidance design on unmanned aerial vehicle in 3d space. Unmanned Syst. 6(4), 277–295 (2018)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Temasek Laboratories at the National University of Singapore, Singapore, Singapore

    Wai Lun Leong, Sunan Huang & Rodney Swee Huat Teo

  2. Department of Mathematics, Computer Science and Physics, University of Udine, Udine, Italy

    Niki Martinel, Christian Micheloni & Gian Luca Foresti

Authors
  1. Wai Lun Leong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Niki Martinel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sunan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christian Micheloni
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gian Luca Foresti
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Rodney Swee Huat Teo
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Perception: Niki Martinel, Christian Micheloni, Gian Luca Foresti are with the Department of Mathematics, Computer Science and Physics, University of Udine, Udine, Italy. Behaviour: Wai Lun Leong, Sunan Huang, Rodney Teo are with Temasek Laboratories at the National University of Singapore, Singapore.

Corresponding author

Correspondence to Wai Lun Leong.

Ethics declarations

Conflict of Interests

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

A short version of this paper was presented in ICUAS 2020.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Leong, W.L., Martinel, N., Huang, S. et al. An Intelligent Auto-Organizing Aerial Robotic Sensor Network System for Urban Surveillance. J Intell Robot Syst 102, 33 (2021). https://doi.org/10.1007/s10846-021-01398-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-021-01398-y

Keywords

Search

Navigation