Non-verbal evaluation of retail service encounters through consumers’ facial expressions

Highlights

• New systems are able to detect consumers' emotions towards a certain retail setting.

• New systems would support employees to better understand consumers' shopping experience.

• Data refers to 28,481 pictures (42,140 faces) from the 19 biggest shopping centers in UK.

• Facial expression recognition systems would uncover consumers’ evaluation of retail service encounters.

• Consumers would accept the usage of facial expression recognition systems to automatically evaluate the retail service.

Abstract

Emotions have been largely acknowledged as important drivers of many consumers' behaviors. They are usually recognized through particular facial expressions, body language and gesture. However, the increasing integration of automatic systems in retailing is pushing researchers to understand the extent to which these systems can support employees to better understand consumers' shopping experience. In this vein, the present research aims at investigating the extent to which it is possible to systematically evaluate retail service encounters through consumers' facial expression. To this end, the research provides a machine learning algorithm to detect the six fundamental (human) emotions based on facial expressions associated with consumers' shopping experience in the 19 biggest shopping centers in UK, and (ii) investigates consumers' response to the usage of this system to automatically collect their evaluation of the retail service encounters. Findings reveal that a facial expression recognition system would uncover consumers’ evaluation of retail service encounters, and that consumers would accept the usage of facial expression identification systems to automatically evaluate the retail service encounters.

Introduction

Recent studies (Huang, Rust, & Maksimovic, in press.) have highlighted the shift of modern society towards the idea of Feeling Economy to embrace the emerging concept of feeling intelligence as the reply to the recent progress in artificial intelligence. Specifically, the authors (Huang et al., in press) defined the Feeling Economy as “a new economy in which the feeling tasks of jobs, such as communicating/coordinating with others and establishing/maintaining interpersonal relationship, are becoming more important for human workers than the thinking tasks of jobs” (p.2). This new perspective may rely on the role of individual's emotions.

Emotions represent a mental state, manifested through particular gestures that are translated into specific actions (Bagozzi, Gopinath, & Nyer, 1999). These are an integrative part of daily life, and constitute an important component of the shopping experience (Babin, Griffin, Borges, & Boles, 2013; Kawaf & Tagg, 2017; Terblanche, 2018), since they influence consumers' behavior, evaluation of products, purchase intention, loyalty, etc. (Frank, Torrico, Enkawa, & Schvaneveldt, 2014; (Gardner, 1985; Lajante & Ladhari, 2019; Kim, Park, Lee, & Choi, 2016b; Malik & Hussain, 2017; Ou & Verhoef, 2017; Das and Varshneya, 2017). Indeed, emotion exchange may affect consumers' brand attitudes, and evaluation of retail services (Wang, 2009). While positive emotions might generate a positive consumers' attachment towards a brand, negative ones might result in negative behavior (i.e. switching, complaining, negative word of mouth, and so on) (Romani, Grappi, & Dalli, 2012). For instance, anxiety and anger experienced during service consumption would lead to dissatisfaction resulting in avoidance behaviors of a certain store (Menon & Dubé, 2004; Otieno, Harrow, & Lea-Greenwood, 2005). Thus, distinguishing consumers' emotions (i.e., positive and negative emotions) would further result in a more effective prediction of the subsequent shopping behavior (Hooge de, 2014; Romani et al., 2012). For these reasons, past studies further suggested to enhance practices to systematically evaluate consumers’ emotion before and after entering the store (Kim et al., 2016b).

However, in modern retail settings, consumers are massively exposed to technology like digital assistants that might influence differently consumers’ behavior (Vannucci and Pantano, 2019; Pantano & Gandini, 2017). Indeed, actual digital assistants are not fully able to execute feeling tasks (Huang et al., in press). In particular, the interaction between machine and human, bounded by interaction protocols and restricted by the embedded information, may both create a different set of emotions that should be attended and challenge traditional consumer-salesforce interaction styles. Thus, new questions arise in the emerging competitive scenario:

RQ1: How can technology support employees to better understand consumers’ shopping experience?

RQ2: To what extent will consumers accept the usage of this system to automatically collect their evaluation of the retail service encounters?

The aim of this paper is to understand the extent to which it is possible to systematically evaluate retail service encounters through consumers' facial expression. In this way, machines would support employees to better understand consumers' shopping experience and reply accordingly. To this end, the research provides a machine learning algorithm to detect the six fundamental (human) emotions based on consumers emotions in non-verbal expressions (i.e., facial expressions) associated with consumers' shopping experience in the 19 biggest shopping centers in UK, and (ii) investigates consumers’ response to the usage of this system to automatically collect their evaluation of the retail service encounters.

The paper is organized as follows: the next section reviews the theoretical background. The subsequent part discusses the studies on emotion recognition. Then, the paper introduces the research method and approach. The paper concludes with the discussion of the main findings, while proposing some suggestions for future studies.