Emergent behavior arising in a joint human-robot system cannot be fully predicted based on an understanding of the individual agents. Typically, robot behavior is governed by algorithms that optimize a reward function that should quantitatively capture the joint system's goal. Although reward functions can be updated to better match human needs, this is no guarantee that no misalignment with the complex and variable human needs will occur. Algorithms may learn undesirable behavior when interacting with the human and the intrinsically unpredictable human-inhabited world, thereby producing further misalignment with human users or bystanders. As a result, humans might behave differently than anticipated, causing robots to learn differently and undesirable behavior to emerge. With this short paper, we state that to design for Human-Robot Interaction that mitigates such undesirable emergent behavior, we need to complement advancements in human-robot interaction algorithms with human factors knowledge and expertise. More specifically, we advocate a three-pronged approach that we illustrate using a particularly challenging example of safety-critical human-robot interaction: a driver interacting with a semi-automated vehicle. Undesirable emergent behavior should be mitigated by a combination of 1) including driver behavioral mechanisms in the vehicle's algorithms and reward functions, 2) model-based approaches that account for interaction-induced driver behavioral adaptations and 3) driver-centered interaction design that promotes driver engagement with the semi-automated vehicle, and the transparent communication of each agent's actions that allows mutual support and adaptation. We provide examples from recent empirical work in our group, in the hope this proves to be fruitful for discussing emergent human-robot interaction.

中文翻译：

---

减轻驾驶员和半自动车辆之间出现的不良紧急行为

基于对个体代理的理解，无法完全预测人机联合系统中出现的紧急行为。通常，机器人行为由优化奖励函数的算法控制，该奖励函数应定量捕获关节系统的目标。尽管可以更新奖励函数以更好地满足人类需求，但这并不能保证不会与复杂多变的人类需求发生错位。在与人类和本质上不可预测的人类居住世界交互时，算法可能会学习不良行为，从而与人类用户或旁观者产生进一步的错位。结果，人类的行为可能与预期不同，导致机器人学习不同并出现不良行为。有了这篇短文，我们声明，要设计人机交互以减轻这种不良的紧急行为，我们需要用人为因素知识和专业知识来补充人机交互算法的进步。更具体地说，我们提倡一种三管齐下的方法，我们使用一个特别具有挑战性的安全关键人机交互示例来说明：驾驶员与半自动车辆交互。应通过以下组合来减轻不良的紧急行为：1) 在车辆算法和奖励函数中包含驾驶员行为机制，2) 考虑交互诱导的驾驶员行为适应的基于模型的方法，以及 3) 以驾驶员为中心的交互设计，促进驾驶员与半自动车辆的互动，以及每个代理的透明沟通” 允许相互支持和适应的行动。我们提供了我们小组最近的实证工作的例子，希望这对于讨论紧急的人机交互是富有成效的。