Predicting mid-air gestural interaction with public displays based on audience behaviour

Highlights

• A 35-days long field study focused on a public display deployment.

• Results show that audience size and behaviour significantly influence user(s) interactions.

• Predictor models are built to forecast users’ interaction duration and distance.

• A visualisation tool is made available to visualise predictions based on audience bahaviour.

• Both the visualisation tool and the predictor model can be adapted to other deployments.

Abstract

Knowledge about the expected interaction duration and expected distance from which users will interact with public displays can be useful in many ways. For example, knowing upfront that a certain setup will lead to shorter interactions can nudge space owners to alter the setup. If a system can predict that incoming users will interact at a long distance for a short amount of time, it can accordingly show shorter versions of content (e.g., videos/advertisements) and employ at-a-distance interaction modalities (e.g., mid-air gestures). In this work, we propose a method to build models for predicting users’ interaction duration and distance in public display environments, focusing on mid-air gestural interactive displays. First, we report our findings from a field study showing that multiple variables, such as audience size and behaviour, significantly influence interaction duration and distance. We then train predictor models using contextual data, based on the same variables. By applying our method to a mid-air gestural interactive public display deployment, we build a model that predicts interaction duration with an average error of about 8 s, and interaction distance with an average error of about 35 cm. We discuss how researchers and practitioners can use our work to build their own predictor models, and how they can use them to optimise their deployment.

Introduction

Interactive public displays can today be found in airports, universities, shopping malls and more. Although it is generally known that users interact with public displays for very short amounts of time, interaction durations vary widely (Davies et al., 2014; Müller et al., 2012; Parker et al., 2018). The same is true for interaction distances, particularly when using interaction techniques such as mid-air gestures (Müller et al., 2012; Nancel et al., 2015). Knowing the expected interaction duration and distance upfront can bring in a plethora of benefits to the different stakeholders of a public display. For instance, if a system is aware that the current situation will result in the user interacting at a certain distance, it can dynamically determine which interaction modality to employ (Dingler et al., 2015b). Furthermore, depending on the expected interaction duration, the system could, for example, dynamically choose which version of an advertisement video to show according to the video length. In addition to run-time benefits, knowing which factors influence interaction duration and distance can be of great value to stakeholders who would then be able to tweak their setups to achieve the optimal user experience.

To address such issues, designers developed methods that try to keep users interacting longer by, for example, showing autopoiesic content (Memarovic et al., 2011), or presenting new content immediately after the user has finished interacting with current content (Alt et al., 2016). Moreover, previous works reported that users interact at different distances, particularly when using at-a-distance interaction techniques such as mid-air gestures (Müller et al., 2012; Nancel et al., 2015). The surrounding environment also affects how users position themselves: for instance, benches surrounding the display potentially result in larger audiences, which in turn discourage users from positioning themselves close to the display (Gentile et al., 2017a). These facts have led researchers to study how to guide users into positioning themselves in the optimal sweet spot (Alt et al., 2015; Zhang et al., 2014).

In contrast to display-centred approaches discussed in previous work, in this paper we focus on the people near the display, and in particular on estimating how the audience could affect possible interactions with the display itself. In more detail, here we present a method for building models to predict the duration and distance of interactions based on the behaviour of users (i.e., people who actually interact with the display) and audience (i.e., people who sit or stay around the display), and on the relationship between them.

To this end, we first analysed the behaviour of users and audience in a real-world deployment of a public display. We chose a deployment in which mid-air gestures were employed for interaction because this modality often results in varying interaction durations (Ackad et al., 2016) and interaction distances (Müller et al., 2012). Data analysis revealed that the users’ interaction duration and distance from the display are significantly influenced by the number of users, the size of the audience, the user-audience relationship, and the audience’s gaze towards the user(s). We then used the collected data to train predictor models based on our method, able to estimate the probability density functions (PDFs) for interaction duration and distance (see Fig. 1), using audience-related information. Our solution estimates the interaction duration with a mean absolute error (MAE) of about 8 seconds, and the interaction distance with a MAE of about 35 cm. We describe our approach in details and discuss how the outcomes of our study, which include the predictor model and a visualisation tool, can be reused by researchers and designers to build predictor models for their public display setups.

We discuss how this work can help space owners and designers to optimise user engagement and experience, e.g., by (a) building similar models and (b) using them to predict how settings will influence users to (c) ultimately optimise the setup. We also explain how to leverage this knowledge to adapt content and interaction modalities in real time.

The contribution of this work is threefold:

• we report our findings from a 35-day long observation of our deployment to identify factors that influence users’ behaviour;

• we propose and evaluate a machine learning approach based on expectation maximisation, aimed at generating predictor models using the collected data;

• we describe how similar models can be developed for predicting user behaviour on public displays.

To help visualise the predictions, we implemented a visualisation tool that can be adapted to other similar deployments. The tool is freely available and open source¹