Autonomous unmanned systems and robots must be able to actively leverage all available information sources — including imprecise but readily available semantic observations provided by human collaborators. This work develops and validates a novel active collaborative human–machine sensing solution for robotic information gathering and optimal decision making problems, with an example implementation of a dynamic target search scenario. Our approach uses continuous partially observable Markov decision process (CPOMDP) planning to generate vehicle trajectories that optimally exploit imperfect detection data from onboard sensors, as well as semantic natural language observations that can be specifically requested from human sensors. The key innovations are a method for the inclusion of a human querying/sensing model in a CPOMDP based autonomous decision making process, as well as a scalable hierarchical Gaussian mixture model formulation for efficiently solving CPOMDPs with semantic observations in continuous dynamic state spaces. Unlike previous state-of-the-art approaches this allows planning in large, complex, highly segmented environments. Our solution is demonstrated and validated with a real human–robot team engaged in dynamic indoor target search and capture scenarios on a custom testbed.

自治的无人系统和机器人必须能够主动利用所有可用的信息源，包括人类合作者提供的不精确但随时可用的语义观察。这项工作开发并验证了一种新颖的主动协作式人机感测解决方案，以解决机器人信息收集和最佳决策问题，并提供了动态目标搜索场景的示例实现。我们的方法使用连续的部分可观察的马尔可夫决策过程（CPOMDP）规划来生成车辆轨迹，以最佳地利用来自车载传感器的不完善检测数据以及可以从人类传感器中特别请求的语义自然语言观察。关键的创新是在基于CPOMDP的自主决策过程中包含人类查询/感知模型的方法，以及可扩展的分层高斯混合模型公式化，以有效地解决具有连续动态状态空间中语义观察的CPOMDP的问题。与以前的最新方法不同，这允许在大型，复杂，高度细分的环境中进行规划。我们的解决方案得到了真正的机器人团队的演示和验证，该团队致力于在定制测试床上进行动态室内目标搜索和捕获场景。高度细分的环境。我们的解决方案得到了真正的机器人团队的演示和验证，该团队致力于在定制测试床上进行动态室内目标搜索和捕获场景。高度细分的环境。我们的解决方案得到了真正的机器人团队的演示和验证，该团队致力于在定制测试床上进行动态室内目标搜索和捕获场景。