Abstract Moving and operating autonomously in a field or orchard environment is challenging for agricultural robots due to the complex task requirements and highly unstructured conditions. The human–computer interaction-based remote control can provide the robots with an alternative solution to assist decision making and motion planning. In this study, a virtual reality (VR) and Kinect-based immersive teleoperation system were proposed to connect the physical and virtual world by utilizing real-time large-scale unstructured agricultural environment reconstruction and simultaneous virtual environment creation. The proposed system, with a relatively large server, can convert the scene into a realistic model by combing the depth and color image streams received from Kinect, and project them back into 3D (three dimensions) space in such a manner that the real 3D scene inside the camera’s field of view is recreated virtually. To create a VR environment for a VR headset, an optimized Bundlefusion-based algorithm was developed for real-time 3D reconstruction of the unstructured agricultural scene in the natural environment. Additionally, the performance of the proposed real-time 3D reconstruction algorithm was evaluated and compared with Bundlefusion and voxel hashing algorithms in different large-scale unstructured agricultural environments. Performing optimizing pose on our optimized algorithm leverages a large number of processing cores available to minimize the delay between data capture and rendering, and it reduces the average acquired time to process each frame no more than 0.9 ms. The experimental results including, less computer storage occupied, fast frame processing time, and high-quality 3D realistic model indicate that our proposed system and algorithm have the potential applicability of immersive teleoperation in an unstructured agricultural environment.

中文翻译：

---

利用 VR 和基于 Kinect 的沉浸式遥操作为农田机器人进行实时 3D 非结构化环境重建

摘要 由于复杂的任务要求和高度非结构化的条件，在田地或果园环境中自主移动和操作对农业机器人来说是一个挑战。基于人机交互的远程控制可以为机器人提供辅助决策和运动规划的替代解决方案。在这项研究中，提出了一种基于虚拟现实 (VR) 和 Kinect 的沉浸式遥操作系统，通过利用实时大规模非结构化农业环境重建和同步虚拟环境创建来连接物理世界和虚拟世界。所提出的系统具有相对较大的服务器，可以通过结合从 Kinect 接收到的深度和彩色图像流，将场景转换为逼真的模型，并将它们投影回 3D（三维）空间，以虚拟方式重建摄像机视野内的真实 3D 场景。为了为 VR 头显创建 VR 环境，我们开发了一种优化的基于 Bundlefusion 的算法，用于对自然环境中的非结构化农业场景进行实时 3D 重建。此外，在不同的大规模非结构化农业环境中，对所提出的实时 3D 重建算法的性能进行了评估，并与 Bundlefusion 和体素哈希算法进行了比较。在我们优化的算法上执行优化姿势利用大量可用的处理核心来最小化数据捕获和渲染之间的延迟，并将处理每帧的平均获取时间减少到不超过 0.9 毫秒。