Drone-surveillance for search and rescue in natural disaster

doi:10.1016/j.comcom.2020.03.012

Computer Communications

Volume 156, 15 April 2020, Pages 1-10

https://doi.org/10.1016/j.comcom.2020.03.012 Get rights and content

Abstract

Due to the increasing capability of drones and requirements to monitor remote areas, drone surveillance is becoming popular. In case of natural disaster, it can scan the wide affected-area quickly and make the search and rescue (SAR) faster to save more human lives. However, using autonomous drone for search and rescue is least explored and require attention of researchers to develop efficient algorithms in autonomous drone surveillance. To develop an automated application using recent advancement of deep-learning, dataset is the key. For this, a substantial amount of human detection and action detection dataset is required to train the deep-learning models. As dataset of drone surveillance in SAR is not available in literature, this paper proposes an image dataset for human action detection for SAR. Proposed dataset contains 2000 unique images filtered from 75,000 images. It contains 30000 human instances of different actions. Also, in this paper various experiments are conducted with proposed dataset, publicly available dataset, and stat-of-the art detection method. Our experiments shows that existing models are not adequate for critical applications such as SAR, and that motivates us to propose a model which is inspired by the pyramidal feature extraction of SSD for human detection and action recognition Proposed model achieves 0.98mAP when applied on proposed dataset which is a significant contribution. In addition, proposed model achieve 7% higher mAP value when applied to standard Okutama dataset in comparison with the state-of-the-art detection models in literature.

Introduction

Search and rescue (SAR) have been a human-intensive task so far, but recent technological advancements can make it autonomous. Using drone surveillance with a recent computer vision technology can increase the number of humans saved at the time of disaster. However, recent activities of using the drone have mixed reaction, but using a drone to save someone’s life is novel and a great cause. Recently, drones are being used more for SAR and providing excellent support in those operations. These days, police and fire departments have also adopted drones and collaborated with local SAR teams for time-sensitive rescue operations. In January of 2019, a SAR team in Snowy Canyon State Park, Utah, used a drone to help rescue a hiker trapped on a ledge at night. The hiker was 60 years old, and SAR first found that he was trapped from other hikers who heard the man calling out for help [1]. Also, a rescue event in Texas, drone was used to find an 88-year-old missing man [1]. In another story of using the drone for SAR where it is used to find two cousins trapped on a mountainside in Iceland [1]. All these incidents show the capability and importance of the drone in the field of SAR however, these operation are performed manually for finding the person which need an automation to apply at bigger scale. These autonomous drone with on-device video analysis capability for saving life motivates us to develop a novel and dedicated system for autonomous searching of people who are stuck and required rescue.

The idea of drone-surveillance for SAR is to use the drone for scanning the affected area with the help of camera, and model deployed on the drone itself for identifying the exact places where help is required. An example of automated surveillance and search operation is shown in Fig. 1. In this figure, after the identification of humans location, GPS location of human can be sent to the rescue team for the fast and productive rescue. The recent success of deep-learning approaches for object detection and action recognition motivates us to apply it in the drone-surveillance. The essential part of a deep-learning approach is that a significant amount of data is needed for training. Since, most of the dataset available in the literature are for ground-level surveillance such as UCF [2], which is not useful for training deep-learning model of aerial surveillance. Hence, it is our primary objective to develop a dataset of aerial action recognition for SAR. In addition, deep-learning models uses these dataset for training in different type of task such as classification and localization. Deep-learning models used for these task can automatically extracts the feature. Out of all other neural networks used for classification or localization, convolution neural networks (CNN) suits more for image-based feature extraction. In CNN, each layer uses a convolution filter for feature extraction. An example of two different type of such task is represented in Fig. 2. In this, detection is a combination of classification and localization. The classification problem of images is mainly to classify the image into a different category (labels), while the objective of detection is to identify the label of the object as well as to determine the exact position of classified labels in that image.

As dataset plays a crucial role in the performance of model, this paper proposes a unique dataset of aerial action recognition for SAR. Also, in the aerial surveillance, since, human appears very small and existing algorithms are not able to identify the action performed by them, this paper also proposes a modified action detection model for aerial action detection. The main contribution of the paper is as follows:

•
In order to develop any application using deep-learning, the primary requirement is availability of labelled dataset. But for automated search of human using drone surveillance there is no such dataset available in literature. Therefore, in this paper, we have proposed a novel dataset to search humans in rescue for disaster management application.
•
Proposed dataset is annotated for two different set of action and is available in the form two action dataset and six action dataset for SAR.
•
In addition, an experimental analysis of deep learning object detection models such as Faster R-CNN, R-FCN, and single shot detection (SSD) applied to existing aerial action detection dataset [3] and proposed dataset, has been presented in the paper. Moreover, an modified SSD has also been proposed for better performance in aerial surveillance.

Section snippets

Related work

Here, we briefly introduce the current work in the field of dataset and the application of model for aerial human action recognition.

Dataset development

This section describes the dataset collection, type of action recorded, pre-processing, and the usefulness of dataset for vision-related real-life applications.

Performance evaluation metrics

mAP and IOU are the standard COCO evaluation parameters used to the evaluate object detection models. Hence, to compare our proposed model with the state-of-the-art models in literature, these parameters are suitable. The details of these parameters are discussed in this section. Also, as precision and recall are two basic parameters on which all these evaluation parameters depends, a brief description of these parameters are also given in this section.

Proposed framework

Proposed architecture with the developed dataset for action detection in drone surveillance, can be used to identify the situations where humans are asking for help. As shown in Fig. 1, on board autonomous analysis of drone images can quickly find humans stuck in the disaster prone area. Proposed dataset is generalized and have enough variation to be used for the automation of such application. In addition, the architecture of proposed model for detecting the action is shown in Fig. 6. Proposed

Experimental setup

Experiments were performed on the NVIDIA DGX-1 V100 supercomputer having 7.8 TFLOP/s for FP64 computation power. To recognize human action, state-of-art object detection was applied to the proposed dataset. In addition to this, experiments were performed on publicly available Okutama dataset as well. Both datasets have frames that contain multiple people performing different actions simultaneously. As the image size of both datasets is equal, i.e., 1920 * 1080 pixels, the results of the various

Results and analysis

In this section, we have discussed the results in detail based on the availability of visual and statistical results for human detection and action recognition.

Table 5 shows the performance of deep learning object detection models applied on publicly available Okutama dataset. The performance is evaluated on a standard coco evaluation metric (mAP). Our result shows that faster R-CNN is performing comparatively better on this dataset. In addition to this, Table 6 shows the results of models

Conclusion

In this paper, we have proposed a drone dataset for human action recognition. This dataset can also be used for human detection and other such task for different surveillance applications. Proposed dataset has a rich amount of variety in terms of colour, height, actor, and background. This variation makes it generalized for proposed dataset to be used for various applications. In addition, as our primary objective is to provide the support for SAR using drone surveillance, we have presented an

CRediT authorship contribution statement

Balmukund Mishra: Conceptualization, Methodology, Writing - original draft. Deepak Garg: Data curation, Investigation, Supervision, Writing - original draft, Project administration. Pratik Narang: Visualization, Investigation, Writing - original draft. Vipul Mishra: Data curation, Formal analysis, Supervision, Writing - original draft, Visualization.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (39)

PisharadyPramod Kumar et al.
Recent methods and databases in vision-based hand gesture recognition: a review
Comput. Vis. Image Underst.
(2015)
DukowitzZacc
Drones in search and rescue: 5 stories showcasing ways search and rescue uses drones to save lives
(2019)
SoomroKhurram et al.
UCF101: A dataset of 101 human actions classes from videos in the wild
(2012)
Mohammadamin Barekatain, Miquel Martí, Hsueh-Fu Shih, Samuel Murray, Kotaro Nakayama, Yutaka Matsuo, Helmut Prendinger,...
Joao Carreira, Andrew Zisserman, Quo vadis, action recognition, a new model and the kinetics dataset, in: Proceedings...
KangSoo Min et al.
Review of action recognition and detection methods
(2016)
SoleimaniAmir et al.
Convolutional neural networks for aerial multi-label pedestrian detection
Asanka G. Perera, Yee Wei Law, Javaan Chahl, UAV-GESTURE: a dataset for UAV control and gesture recognition, in:...
UijlingsJasper RR et al.
Selective search for object recognition
Int. J. Comput. Vis.
(2013)
PurkaitPulak et al.
SPP-NEt: Deep absolute pose regression with synthetic views
(2017)

Ross Girshick, Fast r-cnn, in: Proceedings of the IEEE International Conference on Computer Vision, 2015, pp....

RenShaoqing et al.

Faster r-cnn: Towards real-time object detection with region proposal networks

WangLinzhao et al.

Saliency detection with recurrent fully convolutional networks

Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, You only look once: Unified, real-time object detection,...

LiuWei et al.

Ssd: Single shot multibox detector

FelzenszwalbPedro F et al.

Object detection with discriminatively trained part-based models

IEEE Trans. Pattern Anal. Mach. Intell.

(2009)

SommerLars Wilko et al.

Fast deep vehicle detection in aerial images

QuTao et al.

Vehicle detection from high-resolution aerial images using spatial pyramid pooling-based deep convolutional neural networks

Multimedia Tools Appl.

(2017)

SommerLars et al.

Comprehensive analysis of deep learning based vehicle detection in aerial images

IEEE Trans. Circuits Syst. Video Technol.

(2018)

Cited by (222)

Exploring innovative techniques for damage control during natural disasters
2024, Journal of Safety Science and Resilience
The study critically examines the principles, mechanisms, and effectiveness of different damage control techniques in dealing with natural disasters, emphasizing their pivotal role in minimizing casualties and economic losses. Each of these damage control techniques is mapped based on their applications and relevance in the key areas of natural disaster management. By utilizing various real-world instances, the present study shows that the effective implementation of various innovative techniques is shaping the space of natural disaster management in a global context. The integration of different innovative techniques into the existing natural disaster management system has improved the survival rate, economic performance, and sustainable development. The study finds that innovative disaster financing models, clear strategies, and creating awareness among communities can improve the overall efficiency of innovative techniques that are currently used for damage control during natural disaster events. Despite the substantial advantages of these creative strategies, the study acknowledges challenges such as financial constraints, unclear policy goals, and community adaptation requirements. The study also indicates that in the future, automatic damage restoration, quick prototyping, and additive engineering will play a vital role in controlling damage from catastrophic events, while it acknowledges limitations in temporal scope, generalizability, and financial constraints.
Accurate drone corner position estimation in complex backgrounds with boundary classification
2024, Heliyon
This study develops an efficient approach for precise channel frame detection in complex backgrounds, addressing the critical need for accurate drone navigation. Leveraging YOLACT and group regression, our method outperforms conventional techniques that rely solely on color information. We conducted extensive experiments involving channel frames placed at various angles and within intricate backgrounds, training the algorithm to effectively recognize them. The process involves initial edge image detection, noise reduction through binarization and erosion, segmentation of channel frame line segments using the Hough Transform algorithm, and subsequent classification via the K-means algorithm. Ultimately, we obtain the regression line segment through linear regression, enabling precise positioning by identifying intersection points. Experimental validations validate the robustness of our approach across diverse angles and challenging backgrounds, making significant advancements in UAV applications.
A Machine learning approach for Post-Disaster data curation
2024, Advanced Engineering Informatics
Image data collected after natural disasters play an important role in the forensics of structure failures. However, curating and managing large amounts of post-disaster imagery data is challenging. In most cases, data users still have to spend much effort to find and sort images from the massive amounts of images archived for past decades in order to study specific types of disasters. This paper proposes a new machine learning based approach for automating the labeling and classification of large volumes of post-natural disaster image data to address this issue. More specifically, the proposed method couples pre-trained computer vision models and a natural language processing model with an ontology tailed to natural disasters to facilitate the search and query of specific types of image data. The resulting process returns each image with five primary labels and similarity scores, representing its content based on the developed word-embedding model. Validation and accuracy assessment of the proposed methodology was conducted with ground-level residential building panoramic images from Hurricane Harvey. The computed primary labels showed a minimum average difference of 13.32% when compared to manually assigned labels. This versatile and adaptable solution offers a practical and valuable solution for automating image labeling and classification tasks, with the potential to be applied to various image classifications and used in different fields and industries. The flexibility of the method means that it can be updated and improved to meet the evolving needs of various domains, making it a valuable asset for future research and development.
Global strong convexity and characterization of critical points of time-of-arrival-based source localization
2024, Computational Geometry: Theory and Applications
In this work, we study a least-squares formulation of the source localization problem given time-of-arrival measurements. We show that the formulation, albeit non-convex in general, is globally strongly convex under certain condition on the geometric configuration of the anchors and the source and on the measurement noise. Next, we derive a characterization of the critical points of the least-squares formulation, leading to a bound on the maximum number of critical points under a very mild assumption on the measurement noise. In particular, the result provides a sufficient condition for the critical points of the least-squares formulation to be isolated. Prior to our work, the isolation of the critical points is treated as an assumption without any justification in the localization literature. The said characterization also leads to an algorithm that can find a global optimum of the least-squares formulation by searching through all critical points. We then establish an upper bound of the estimation error of the least-squares estimator. Finally, our numerical results corroborate the theoretical findings and show that our proposed algorithm can obtain a global solution regardless of the geometric configuration of the anchors and the source.
Multi-objective reinforcement learning for autonomous drone navigation in urban areas with wind zones
2024, Automation in Construction
Drones can be used for tasks such as data collection and logistics in civil engineering. Current research on autonomous drones mainly focuses on planning a safe path and avoiding obstacles in a static environment. However, navigating a drone in complex environments like urban areas involves many dynamic constraints, such as building layout, winds, and signal coverage, which are interdependent. The wind factor is the most important among these environmental factors, which may cause a drone to lose control or even crash. This paper presents a multi-objective navigation reinforcement learning algorithm (MONRL) for the drone to navigate and avoid obstacles in an unknown environment when dynamic wind zones present, with only imagery data about the building layouts. Based on a deep reinforcement learning and memory architecture, the drone develops policies to prioritize navigation decisions, optimizing the path while minimizing negative impact of winds with only sparse sensor data, in our case, camera inputs. By leveraging the advantages of the proposed method in estimating the environmental factors from previous trials, no aerodynamic force sensors are needed for the drone to develop effective strategies to navigate to target while counteracting to milder winds and dodging aways from stronger winds. The proposed method was tested in a virtual environment, and a real model of New York City. The method is expected to contribute to new automation algorithms for urban aerial logistics and future automated civil infrastructure inspection.
Exploring artificial intelligence for applications of drones in forest ecology and management
2024, Forest Ecology and Management
This paper highlights the significance of Artificial Intelligence (AI) in the realm of drone applications in forestry. Drones have revolutionized various forest operations, and their role in mapping, monitoring, and inventory procedures is explored comprehensively. Leveraging advanced imaging technologies and data processing techniques, drones enable real-time tracking of changes in forested landscapes, facilitating effective monitoring of threats such as fire outbreaks and pest infestations. They expedite forest inventory by swiftly surveying large areas, providing precise data on tree species identification, size estimation, and health assessment, thus supporting informed decision-making and sustainable forest management practices. Moreover, drones contribute to tree planting, pruning, and harvesting, while monitoring reforestation efforts in real-time. Wildlife monitoring is also enhanced, aiding in the identification of conservation concerns and informing targeted conservation strategies. Drones offer a safer and more efficient alternative in search and rescue operations within dense forests, reducing response time and improving outcomes. Additionally, drones equipped with thermal cameras enable early detection of wildfires, enabling timely response, mitigation, and preservation efforts. The integration of AI and drones holds immense potential for enhancing forestry practices and contributing to sustainable land management. In the future explainable AI (XAI) improves trust and safety by providing transparency in decision-making, aiding in liability issues, and enabling precise operations. XAI facilitates better environmental monitoring and impact analysis, contributing to efficient forest management and preservation efforts. If a drone's AI can explain its actions, it will be easier to understand why it chose a particular path or action, which could inform safety procedures and improvements.

View all citing articles on Scopus

View full text