Firefighters need to gain information from both inside and outside of buildings in first response emergency scenarios. For this purpose, drones are beneficial. This paper presents an elicitation study that showed firefighters’ desires to collaborate with autonomous drones. We developed a Human–Drone interaction (HDI) method for indicating a target to a drone using 3D pointing gestures estimated solely from a monocular camera. The participant first points to a window without using any wearable or body-attached device. Through the drone’s front-facing camera, the drone detects the gesture and computes the target window. This work includes a description of the process for choosing the gesture, detecting and localizing objects, and carrying out the transformations between coordinate systems. Our proposed 3D pointing gesture interface improves on 2D interfaces by integrating depth information with SLAM and solving ambiguity with multiple objects aligned on the same plane in a large-scale outdoor environment. Experimental results showed that our 3D pointing gesture interface obtained average F1 scores of 0.85 and 0.73 for precision and recall in simulation and real-world experiments and an F1 score of 0.58 at the maximum distance of 25 m between the drone and building.

中文翻译：

---

3D指向手势作为目标选择工具：在室外环境中的窗口选择期间指导单眼无人机

在第一响应紧急情况下，消防员需要从建筑物内部和外部获取信息。为此，无人机是有益的。本文提出了一项启发性研究，该研究表明了消防员与自动无人机合作的愿望。我们开发了人机交互（HDI）方法，使用仅从单眼相机估计的3D指向手势将目标指示为无人机。参与者首先指向窗户，而不使用任何可穿戴或与身体连接的设备。通过无人机的前置摄像头，无人机可以检测到手势并计算目标窗口。这项工作包括对选择手势，检测和定位对象以及执行坐标系之间的转换的过程的描述。我们提出的3D指向手势界面通过将深度信息与SLAM集成在一起，并解决了在大型室外环境中在同一平面上对齐的多个对象的歧义，从而改进了2D界面。实验结果表明，我们的3D指向手势界面在模拟和真实实验中的精度和召回率方面分别获得0.85和0.73的平均F1分数，在无人机与建筑物之间的最大距离为25 m时，其F1分数为0.58。