Combining gaze and AI planning for online human intention recognition

Abstract

Intention recognition is the process of using behavioural cues, such as deliberative actions, eye gaze, and gestures, to infer an agent's goals or future behaviour. In artificial intelligence, one approach for intention recognition is to use a model of possible behaviour to rate intentions as more likely if they are a better ‘fit’ to actions observed so far. In this paper, we draw from literature linking gaze and visual attention, and we propose a novel model of online human intention recognition that combines gaze and model-based AI planning to build probability distributions over a set of possible intentions. In human-behavioural experiments ($n=40$) involving a multi-player board game, we demonstrate that adding gaze-based priors to model-based intention recognition improved the accuracy of intention recognition by 22% ($p<0.05$), determined those intentions ≈90 seconds earlier ($p<0.05$), and at no additional computational cost. We also demonstrate that, when evaluated in the presence of semi-rational or deceptive gaze behaviours, the proposed model is significantly more accurate (9% improvement) ($p<0.05$) compared to a model-based or gaze only approaches. Our results indicate that the proposed model could be used to design novel human-agent interactions in cases when we are unsure whether a person is honest, deceitful, or semi-rational.

Introduction

Autonomous software agents and robots are becoming part of modern society [1], and therefore, the development of autonomous agents that can function as productive members of human-agent teams becomes ever important. To achieve effective interactions with humans, artificial agents must reason about the goals, intentions, and beliefs of other agents [2]. By observing the history of other agents' actions—such as physical actions or verbal utterances—agents can build a basis for recognising intentions and predicting future actions, which in turn shapes their interactions with these agents.

Recent work has successfully used automated planning [3] for model-based intention recognition of intelligent agents. These approaches rely on sequences of already performed physical (ontic) actions [4], [5] for making their plans and project possible future actions to predict intentions. Ontic actions are actions that modify the state of the world. In contrast to the popularity of ontic actions for building intention recognition models, nonverbal signals such as gaze have been relatively under-explored. Gaze is a crucial signal in human nonverbal communication and, as such, offers promising directions for enhancing interactions in human-agent teams [6] and improving intention recognition [7], [8], [9]. With decreasing cost and increasing robustness, eye trackers are entering the consumer market. Eye movements play an important role in planning and executing actions and intentions [10]—both in the short- and long-term. Eye tracking is used to understand decision making [11], improve learning and training systems [12], [13], [14], [15], in human-robot interactions [6], [16], and games [17]. Agents can monitor humans' eye movements to anticipate future actions [7], [9] and to determine their level of engagement [18], [19]. Researchers now have a better understanding of how to link eye-tracking and attention [11], [20], [13], [21], [22], [23]. These investigations suggest that there is huge potential for intelligent agents to use gaze implicitly to derive people's intentions and adapt the interaction accordingly.

However, there are limitations with existing approaches using gaze for intention recognition. First, machine learning dominates existing approaches [24], [25], [26]. Though successful, these models require sufficient data to train such models — something that we do not have in many of our applications. Second, the models learnt by machine learning algorithms limit the prospects of explaining the reasons behind the inferences due to their opaqueness. Finally, most models have been evaluated in contexts requiring prediction of a single intention, for example, [7], [24], and these intentions are usually short-term or proximal intentions.

In recent work [9], we propose a novel intention recognition approach that incorporated visual behaviour into model-based intention recognition and demonstrate how it substantially improved the recognition performance. In the current paper, we improve this model in three ways. First, we improve the model to include a more robust and realistic model of visual attention and different forms of foveal vision. Second, we extend the model to handle distal intentions, rather than just proximal intentions. Third, we extend the evaluation to show that our model is robust in that it overcomes semi-rational gaze behaviours due to non-task related gaze data, including deceptive data.

Fig. 1 shows a simple example of our model. In this example, based on the board game Ticket to Ride described in Section 4, the player is trying to navigate a path (e.g. direct a vehicle) between Santa Fe and one of the other cities in the graph. The intention recognition problem is to determine the destination city. On the left, we see that the route from Santa Fe to Denver has already been traversed. We argue that this implies that the probability of the final destination being Oklahoma is smaller than that of any other of the other nodes: a rational navigator would more likely traverse the path from Santa Fe to Oklahoma directly. Existing model-based approaches would rate this as such. However, from this single traversed route, we are unable to distinguish the probability of the outer nodes (Seattle, Calgary, Winnipeg, etc.)—they are all the same distance from Denver. However, consider the example on the right, in which we know that the person has been looking at the route from Helena to Seattle. We argue that now, this represents a potential future action and that Seattle is a more likely final destination than Calgary, Winnipeg. We argue further that Calgary, are still more likely than Oklahoma, which fits neither our observed navigation actions nor our observed gaze actions.

We evaluated our model through a human-behavioural study. The study involved 40 players playing a digital multi-agent board game while their gaze data was being recorded. We compared a model that used gaze data only (GazeOnly), a model-based approach using only in-game ontic actions (OnticAction-Only), and a model enhanced with gaze-based priors (Gaze+ OnticAction), and we evaluated the three models with the presence of uncertain (deceptive) gaze. We find the proposed Gaze+OnticAction model outperforms the other two at recognising both proximal and distal intentions and maintains this performance irrespective of gaze behaviour being natural or deceptive. We also demonstrate the potential of the enhanced gaze model to help understand the different forms of foveal vision [27]. The results provide evidence that our combined approach using gaze and ontic action for intention recognition of human behaviour is robust enough that it could be used to improve interaction design between agents and humans in cases with uncertain gaze data. Further, extending to cases in which a person is honest (stable gaze data), deceitful (highly uncertain gaze behaviour), or somewhere in between (semi-irrational gaze behaviour).

The key contributions of this work are as follows:

• A computational model of gaze for intention recognition inspired by existing research on eye-tracking and visual attention.

• An empirical evaluation of the model demonstrating the success of the model at recognising both distal and proximal intentions.

• An empirical evaluation of the model in the presence of uncertain gaze behaviour.

• A model appropriate for situations where designers do not have sufficient data to train intention recognition models and require transparent inferences with better chances of explaining the intention recognition related decisions.

Broadly, our experimental results show that gaze actions are intentional, and therefore, an indicator of humans task-related intentions. Harnessing this ability can help improve human-agent interactions by assisting agents to reason about the human counterparts quicker and more accurately, giving the agents the ability to improve its proactiveness. We provide a theoretical model for combining gaze within AI-planning based intention recognition approach, and a computational model of gaze that can be used to investigate links between eye-tracking and visual attention.