Operating a remote robot in an unknown environment is a challenge. Usually, a camera is attached to the robot to capture the images of the remote scenes. However, it is difficult to perceive a 3D environment using a 2D display. In this study, a real-time simultaneous localization and mapping (SLAM)-based virtual reality (VR) 3D human-machine interaction system is developed to provide users with immersive telepresence to better operate a remote mobile manipulator in an unknown environment. An RGB-depth camera is mounted on the mobile platform to scan and reconstruct the remote environment in real time. A head mounted display is used to display the 3D reconstructed environment, with a virtual copy of the actual robot in it, to provide users the feeling of immersion at the remote site. Visual aids are added in the 3D reconstructed environment to guide users with the direction and distance information. Users can control the remote robot by manipulating the virtual copy of the actual robot in the VR environment, without being constrained in the robot first person view. The system is compared with a video-based teleoperation and a direct on-site operation. Experimental results show that the grasping error and placing threshold of the SLAM-based VR teleoperation are 0.5 cm and 1.5 cm, respectively. The average error of the reconstructed environment is less than 4%. Within the accuracy of the reconstructed environment, the SLAM-based VR teleoperation has a higher grasping success rate and lower operation time, compared to the video-based teleoperation. This study shows a promising application of immersive SLAM-based VR in interactive telerobotic operations in a remote unknown environment.

在未知环境中操作远程机器人是一项挑战。通常，机器人会安装一个摄像头来捕捉远程场景的图像。然而，使用 2D 显示器很难感知 3D 环境。在这项研究中，开发了一种基于实时同步定位和映射 (SLAM) 的虚拟现实 (VR) 3D 人机交互系统，为用户提供身临其境的远程呈现，以便在未知环境中更好地操作远程移动机械手。移动平台上安装了一个RGB深度摄像头，实时扫描和重建远程环境。头戴式显示器用于显示 3D 重建环境，其中包含实际机器人的虚拟副本，为用户提供身临其境的感觉。在 3D 重建环境中添加了视觉辅助工具，通过方向和距离信息来引导用户。用户可以通过在 VR 环境中操纵实际机器人的虚拟副本来控制远程机器人，而不受机器人第一人称视角的限制。将该系统与基于视频的遥操作和直接现场操作进行比较。实验结果表明，基于SLAM的VR遥操作的抓取误差和放置阈值分别为0.5 cm和1.5 cm。重建环境的平均误差小于4%。在重建环境的精度范围内，与基于视频的遥操作相比，基于SLAM的VR遥操作具有更高的抓取成功率和更低的操作时间。