1 Introduction

This paper explores the notion of touch through interviews with designers, developers and researchers working in immersive virtual reality (VR) to better identify how touch ‘connection’ is realised in virtual spaces, through various interactive devices, and how the digital remediation of touch shapes the sociality of touch experiences and touch practices.

Touch landscapes are being differently remediated by digital technology. VR contexts bring particular challenges for designing and developing forms of touch interaction, given VRs historical, technically driven emphasis on the visual and aural. Touch is, however, of great interest to VR due to its central role in how we experience and understand the world, ourselves and others (Bull et al. 2006), and in fostering feelings of connection and reality (Spence and Gallace 2013). A key consideration of VR is its immersive qualities—the experience of spatial presence in the digital environment where media contents are perceived and treated as real (Madigan 2010). A central motivation is to create a sense of ‘connection’ to evoke feelings of presence and enhance user engagement. Designers and developers are grappling with how to improve connection through technical ‘immersive’ capacities including touch. While the value of touch in mediating experiences in VR is increasingly recognised, little research has explored developers’ current conceptions, uses and challenges of touch in VR, or attended to the social implications of touch interaction and communication that are important for development of touch-based VR experiences.

Technology is not value-free (Welchel 1986), development being framed by specific values around ‘touch’-in VR perhaps this is the tactile physical sensation. Acknowledging that touch has strong social and material dimensions, looking beyond the physiological experience, we argue for the need to explore instantiations and developments of touch in immersive VR from a social perspective. Design consideration of the social, political and ethical challenges raised by digital touch (e.g. privacy, safety and digital exclusion) is necessary to support the development of digital touch devices, systems and environments that take account and care of people’s social contexts (Jewitt et al. 2020), and the socially oriented drivers underpinning this development, such as increased distant relationships promoting the need for remote communication opportunities and shared, connected experiences through new touch practices.

Different design responses to the digital mediation of touch have been emerged including mid-air haptics, vibrotactile, exoskeletal and electromechanical solutions. These techniques are integrated into hardware like gloves, vests or suits, or controllers to support interaction in VR, in conjunction with headsets for immersive experiences. Touch is challenging in virtual contexts, since the physical materiality present in our everyday touch interactions with people and objects is not readily present or easily achievable. Indeed, voices challenging the capacity of these technologies to bring touch experiences to VR highlight the complexity of touch at a physiological level (Abrash 2015) and question the maturity of these technologies to support touch experiences outside laboratory settings (Stone 2019). The limitations in touch-eliciting devices mean that VR experiences draw on people’s social and sensorial experiences and expectations of touch through a variety of means, including sound and visualisation. Acknowledging the complexity of exploring touch communication and sensory experience in VR, this paper aims to examine conceptual perspectives of touch in VR. It seeks to engage with current rhetoric underpinning the design and development of digital touch communication experiences, and the implications for future development from a social, sensory and communication perspective. In doing so, we identify how touch is currently being conceptualised in VR and the design impacts of this to examine the potential for generating new forms of touch or changing communicative capacity and to enhance socially orientated understandings, for research and design of digital touch.

We interviewed technology developers, designers, artists and researchers working in the area of virtual touch technologies. Drawing on phenomenological perspectives, with ‘virtual touch’ or digital touch in VR being the phenomenon under study, touch is framed as being part of the wider sensorium in interaction and communication. The complexity of defining ‘virtual’ touch, since much digital touch could to some extent be couched as virtual, led to a reorientation of our initial focus on ‘immersive virtual reality’ using headsets and controllers to ‘touch’ in virtual spaces, to accommodate mid-air haptics where touch is ‘virtual’ in the sense that it is a non-physical contact form of touch. For the purposes of this paper, our notion of virtual touch focuses on experts working in ‘virtual environments’, that include immersive VR, gesture-based VR interfaces, mid-air haptics and ‘touchless’ interactive spaces. We see this as distinct from augmented reality, as physically integrated or overlaid with digital experiences, or mixed reality, which are beyond the scope of this paper.

2 Background

This section foregrounds current developments of touch in VR and the social considerations on touch and technology development.

2.1 Digital touch in VR

Studies point to general benefits of VR, such as promoting brain neuroplasticity (Cheung 2014), rehabilitation (Hsiao-Chinga et al. 2017; Ravi et al. 2017), training (Alaker et al. 2016) and education (Kavanagh et al. 2017). The integration of touch into VR is in the early stages of research development (Stone 2001, 2019) often implied through visual or audio, and unsurprisingly, there are few studies focused on the contribution of touch to these general VR benefits. Nonetheless, the qualities that touch brings to our everyday experience are increasingly seen as important for enhancing a sense of presence (e.g. Campbell et al. 2018) and immersion in VR (Muthukumarana et al. 2019) by making VR tangible in ways vision cannot (Paterson 2006), “cementing a physical link between the user and the virtual world” (Parisi 2018), and “making things “real” to us” (Spence and Gallace 2013).

Touch is integrated or implied in different ways in VR. One approach is to use the virtual hand metaphor, where optical sensors reflect the movement of the hand or map to a visualisation of the hand in VR (Pietroszek 2018). Here, touch illusion relying on visuals and sound is thought to increase immersion. Various digital artefacts being developed also convey touch to enrich VR experiences including gloves (e.g. HaptX, Gu et al. 2016); enhanced controllers through attached vibrotactile motors (e.g. Lee et al. 2019) or mechanically actuators enabling users to feel the shape of virtual objects (Benko et al. 2016); tactile sensations on the hand using mid-air tactile stimulation (Pittera et al. 2019); ‘plasters’ using SMA technology (shape-memory alloys) “recreating the perception of a touch sensation on the forearm (e.g. gentle touch, caressing, clenching or tapping)” (Muthukumarana et al. 2020 p.3); or a full-body haptic suit deploying electrical muscle stimulation (EMS). With such devices, touch is discussed in terms of ‘re-creating touch’ or ‘touch illusions’ (Muthukumarana et al. 2019), the focus of development being on the system or device and what this can achieve in terms of physiological response and/or user interpretation of physical sensation. Thus, VR systems often try to “replicate touch sensations in the most sensitive part of our body, the hands” (Cranny-Francis 2013) or the feet (e.g. Rovers and van Essen 2006), which highlights three challenges for VR. Firstly, the notion of replication gives rise to tension between the opportunities VR brings to experience a ‘touch’ which may be at odds with our everyday touch experiences (e.g. pressing a button on the controller to poke something or stroke something). Secondly, it brings attention to touching versus being touched-controllers, for example, enable the act of ‘touching’ in some way, while a sense of being touched, especially across the body, is not easily enabled. Thirdly, it highlights a focus on body parts—primarily the hands—as a locus of touch in VR.

2.2 The sociality of virtual touch

“We are always, already touching…We are, at all times, touching and being touched” (Cranny-Francis 2013, p.4). Touch is central to our everyday experience, understanding of the world, ourselves and others. Touching provides us with significant information about our world (Finnegan 2014), communication with others (Hernstein 2009) and is beneficial to one’s own well-being.

Touch is a cultural practice imbued with social norms concerning who can legitimately touch what and who, where, how and when. Current VR touch is primarily associated with ‘things’ rather than ‘others’, framing the particular kinds of touch that are under consideration, and which, in common with cinema, is centred around the visual: these ideas being further developed in haptic media studies (Parisi et al. 2017). In pseudohaptics, cross-modal illusions are used to simulate sensory feeling through combined visual and auditory feedback (Collins and Kapralos 2019). Nevertheless, VR needs to go beyond this to engage with touch and kinaesthetic experiences as located on the body more broadly. The body is central to touch experiences, yet again the focus of contemporary VR on hands means it is often overlooked. Relocating virtual touch on the body would raise questions around what kinds of bodies are considered in its design (Jewitt et al. 2020). For example, Macpherson shows how blind participants “drew attention to their feet–transcending the stereotype of blind touch as primarily associated with the hand” (2009, p. 179).

These aspects speak to Welchel’s (1986) notion of the ‘non-neutrality’ of technology and the consequences for framing reality according to value systems placed on technical development. This framing places certain aspects, features or characteristics of a technology over and above others. An emphasis on ‘re-creating touch’ or ‘touch illusions’ (Muthukumarana et al. 2019) suggests VR values reproducing physical touch sensations. However, our everyday experiences of touch go way beyond physical sensations, drawing on cultural, historical and social factors. This brings the importance and desirability of a social lens in the process of technological development. “Technology needs outside [social] forces driving and shaping it; we need independent inputs to spur creativity, guide our direction and provide inspiration” (Welchel 1986 p.6).

3 Methodology

3.1 Methodological approach

In this qualitative case study, we take a phenomenological approach to virtual touch (Simon and Goes 2013) which aims to both sense and make sense of participants’ ‘experiencing’ and perspectives of a phenomenon (Finlay 2013): in this case digital touch design and development. The study aims to access what is directly perceived and focuses on understanding how human beings experience their world (Sutton and Austin 2015), here the focus being on designers and developers’ experiences of digital touch and touch technologies in VR. This approach aligns well with a case study of current design and developer perspectives on digital touch technologies as a bounded system and bounded by time and place (Creswell 2003). The analysis involves an interpretive approach (e.g. van Manen 1991), alongside which we take a transformative research approach (Mertens 2003) to identify social issues and debates arising from technology design and development with a view to better informing future designs that are socially aware.

3.2 Participants

Ten participants were recruited to represent the scope of the current state of the art in VR touch using purposeful sampling, on the basis of being expert technical designers/developers (from design to engineering) from industry and academia, working in the field of VR, with a special interest in virtual touch and typically involving a form of haptic technology in virtual spaces. Participants were identified via the wider InTouch research project (https://in-touch-digital.com), but excluding any direct collaborators. They were selected on the basis of their knowledge of leading edge technology, that embraces a range of VR interaction devices and contexts of application, which included health, education, artistic experiences. Since VR is targeted at those over thirteen years of age, conversations in general assumed an adult audience. Three participants were female, and seven were male, five from research/academia and five from industry (Table 1). While we discuss the sociocultural aspects of touch, this issue is explored at a general design and engineering level rather than in relation to individual interviewee’s sociocultural backgrounds.

Table 1 Participant profiles
Full size table

3.3 Interviews

In-depth semi-structured interviews aimed to prompt participants to talk about their concrete experience, produce narratives about their technology and the role of touch within, engage with a bodily/sensory perspective around virtual touch and where possible to demonstrate their systems. Interviews were designed to support interviewees to contribute their ideas on key topics, and the interviewer to probe further. Questions were structured around the following key themes: participant professional background (role, discipline, interests); description of ‘touch’ in the context of participants’ technology/work; the types of touch enabled in their technical applications; contexts of use; changing notions of touch and the social uses supported; ethical aspects; and future aspirations for touch design.

3.4 Procedure

All participants were invited to participate by email. On agreement, participants were sent an information sheet and consent form, and an interview date was arranged. Interviews were conducted by two researchers and lasted between 50 and 60 min: three undertaken online due to location of interviewee, seven undertaken face-to-face, with three of these providing demonstrations of their technology. Interviews were video recorded to capture multimodal forms of communication, including talk and gesture, which was considered critical given the aim to convey ideas and contexts of touch. Field notes complemented the video data. All participants were given the opportunity to read the article, to ensure accurate reporting of the interview data.

3.5 Analytical approach

A phenomenological approach frames the thematic analysis (Sutton and Austin 2015), to understand the meanings that participants ascribe to their experiences and interpret this within the context of the current conception of virtual touch and societal implications for design and development. Interviews were transcribed time marked and annotated using ELAN to include talk and gestures, like stroking motions. The data were coded and organised into themes that emerged across the data using NVivo (Braun and Clarke 2006), guided by the research aims and the overarching ‘frame of attention’ on touch as a social and sensory experience. Specifically, the analysis engaged with how touch is enabled or evoked in VR, how touch is simulated, the material qualities of touch, where and when touch takes place, opportunities and challenges provided by technology and the degree to which touch in VR remediates touch experiences. This approach led to two overarching themes: how touch is conceptualised; and how the body is brought into the touch experience.

In the next section, we introduce the interactive devices and contexts most pertinent to each of the participants’ work. This lays the foundation for the key technical challenges encountered, as well as the emergent themes around how ‘touch’ is conceptualised in VR spaces: touch as replication, touch as illusion and body fragmentation. We then present and discuss each theme with particular attention to the social implications of touch in VR, which offers insight into key challenges for design and development and social considerations.

4 Technical and contextual shaping of touch

This section presents the techniques, tools (controllers, gloves, gesture-based interface, head tracking) and systems that participants used to mediate touch in virtual spaces and their primary use contexts.

4.1 Haptics

Haptics covers a range of techniques (vibration, force feedback systems and inflating sacs) that are integrated into technological devices that vary considerably in complexity and expense. Despite these variations, the importance of ‘haptics’ is in its ability to generate some form of touch, seen as a ‘must have’ for VR [05].

simple vibrating devices …are remarkably effective …in that you feel something, you feel a vibration, it's completely inaccurate. …. but it alerts you that you have done something, you bumped into something [02].

This sense of touch in VR is perceived to enhance immersion [08, 06, 02], with vibration perceived to convey or increase a user’s feeling of presence [06] and bring ‘more believability’ [02]. Often vibration is implemented using motor feedback or force feedback providing a sensation of pressure which acts as an indicator that you have touched something, that something physical is there and is typically presented in conjunction with visual solidity of an object. Such vibration is implemented through controllers (similar to vibration in devices such as Nintendo Switch) and glove-like devices, but the specific device differently shapes the perceived touch experience, given the different forms of gestural or action interaction elicited with the VR objects.

If I grab this glass, my hand doesn't go through it like its a hologram, it stops round the edges, there’s a stiffness there. That combination of the force feedback–that stiffness, the resistance–and … that pressure sensation–the combination of those things are present in most of the interactions we do [08].

Touch through vibration was viewed as a binary state—touching or not touching—a short temporal event that precludes the use of other tactile cues. Thus, touching a piece of paper or an apple would feel the same through vibration, suggesting a uniformity to digital touch sensation. Their differences are audibly and visually created and situate the touch experience in a wider social and sensory context. Vibration is nevertheless aiming to ‘stand in for’ some form of physical touch, bringing discussion around how to provide more refined forms of feedback using different hardware to generate tactile sensations such as force, edges, resistance and pressure.

According to Abrash (2015), “Haptics is at the core of the way we interact with our surroundings, and without it we will never be fully embodied in a virtual world” (cited Parisi 2018). Although several participants noted that the rendering of haptics to be “good enough to fool the user” was sufficient [e.g. 05], participants highlighted two reasons haptics are perceived as a challenge. Firstly, since we sense touch all over the body, and having systems that enable such broad tactile sensation is very complex [02]. Secondly, the temporal mapping of haptic to visual or audio is critical (02), yet in VR haptics are latency intolerant, although 5G was raised in terms of addressing latency in the haptic range [06, 05].

Social touch in the sense of presence between people really requires low latency. Next generation global networks will have extremely little latency, like 1–10 ms, and that is now in a haptic range so that you can do things like touch a person in real time over the network. That's never been possible before [06].

4.2 Controllers

Controllers, currently the most common device for interacting in VR, featured centrally for two participants but mentioned by the majority. Although these devices were initially designed for manual (hand and finger) interaction in space and with objects, “just a form of keyboard” [05], rather than for eliciting touch sensations per se, participants related ‘virtual touch’ to controller interaction.

Controllers without six DoF used in educational VR were noted as being similar to a touch screen, like dragging your finger to move objects, with limited flexibility making precise touch challenging and fine-grained movements problematic. The reduced ‘touch’ sensitivity of the controller impacts interaction, specifically changing a user’s way of moving or touching: users have to learn how to ‘touch’ through the physical controller. But since fine motor actions in the physical world are not typically needed in VR users can ‘cheat’ the system, adopting alternative interaction movements. Often touch information is not coherent with what the user sees, for example, a press on the controller is less prolonged than a continuous grip on an object [O1]. This realises a ‘temporal’ and ‘binary’ attribution to touch interaction, which can create tensions across touch sensations. While controllers vary in their capabilities, ‘touch’ experiences with controllers were generally perceived to be limited.

Touch is realised through refined force feedback, where the specific hand grip and angle of approach are important in generating different ways of touching. In the context of dentistry training, for example, the perception and interpretation of touch form the foundation of decision making. Here, an OmniPhantom—a specific form of controller with six DoF–enabled the design of the virtual environment to determine these different ways of touching and elicit relevant touch perceptions. Digitisation was noted to provide touch with flexibility, replicability and variety compared with limitations of ‘real world’ dentistry training. Virtual touch tools, practices and sensations were intimately tied to the social and professional expectations of touch (e.g. dentistry). These tools are used to interact in the virtual world guide movement and gesture, and shape and are shaped by the forms of touch elicited and perceived which are rooted in social context.

4.3 Glove-like devices

Three participants focused on glove-like means of engaging touch, aiming to move towards more ‘realistic’ touch interaction by allowing more ‘natural’ movement with hands and fingers. While ‘glove’ design and development is in its early stages and not yet commercially available, VR interaction experiences and professional or industrial training that require specific hand actions, are a future focus.

Go-touch VR comprises three sensor-based devices that attach to the thumb, forefinger and middle finger and provide feedback that corresponds to stiffness, texture and vibration. These devices aim to foster a feeling of ‘active touch’ when grasping something. A holding action maps to a feeling of stiffness and pressure which is considered more realistic, particularly where specific hand and finger actions are critical.

A recent development from HaptX is a glove design using microfluidics to generate touch sensations through the fingertips and over the palm [08].

We run it [compressed air] through teeny tiny channels (where the micro is)–that are like capillaries. They send the air very quickly and we inflate tactile actuators–we call them ‘tactors’, which is kind of like a tactile pixel. We have 130 tactile pixels on each hand and they inflate proportionally when you touch an object. ….We had 12 different zones on your hand for thermal feedback where you could have a hot cold on one part of your hand and a snowball on another part and you could feel both hot and cold simultaneously [08].

Their aim is to make touch more realistic through pressure sensation by causing indentation on the finger and force feedback in terms of a sense of stiffness. Given that the combination of these features are present in most of our touch interactions the gloves aim to create a ‘natural’ touch. Interestingly, the use of ‘tactor’ in the above quote points to the challenge of a lack of standardisation of descriptive language for the conceptualising of touch in VR—an issue we return to later.

Motion tracking from sensors is critical for simulating and inferring touch with glove-like devices. Much effort is placed on accurately modelling visualisations of different hands (e.g. male female) and combining this with touch sensors aims to remove the disconnect often experienced with non-mapped hands. Early research for one participant engaged with gaming experiences that generated specific tactile sensations: feeling the rain, a little fox walking on your hand. The integration of touch in this context is concerned with enriching the experience, whereas current development targeting industrial applications for training and design (e.g. outside VR with telerobotics) involves practising repetitive tasks that require dexterity and muscle memory (e.g. where safety is key for electricity line workers), or in a design context (e.g. a partnership with Nissan) requires more realistic touch interaction and feedback about driver experience through a VR car model. This use of touch, beyond function, echoes the affective, tangential and ephemeral qualities of much social touch.

4.4 Gestural-based interaction

Some VR developers are investigating how touch might be conceived and implemented without haptic feedback and notions of physical materiality, which demands other ways of introducing touch sensations. Several participants noted the use of gestural interfaces that rely on hand tracking as an alternative approach [01, 03, 04, 05, 07, 10], with three emphasising the role of movement and gesture as being instrumental in both shaping and perceiving touch.

One participant designed and developed VR experiences for rehabilitation purposes [03]. VR was considered beneficial since there are no limitations (e.g. no weight on objects) for patients with poor strength, practice can take place beyond clinical environments, and the quality of ‘immersion’ is positive in maintaining motivation and engagement. This is particularly interesting from an accessibility perspective given the typical limitations of VR caused by access to equipment. The participant’s rehabilitation experiences use Leap Motion, keeping the hands free, and touch sensations are visually mediated through simulation tightly linked to accurate motion tracking. The design fosters specific movements of the hand or arm, carrying out tasks that inherently involve touch, such as, picking up, holding and then releasing an object.

When you are doing functional tasks, and that is where we focus, it always engages an object, and interaction with that object one way or the other through touch [03].

Movement and gesture are seen as a constituent part of touch even without haptic feedback. Fostering appropriate movement and gestural positioning in relation to simulating the realism of the object is central: “you would hold and move your hand differently if the object represents a bowl or if the object represents a brick”. If the object is not properly touched or grasped, then it does not move.

A tight coupling of movement and gesture to visualisation is critical, since the effort spent by the patient is thought to be different if they believe they are actually engaging with particular objects and properties that shape their ‘grasp’ and action, “so that the immersion and interaction with the objects are similar to the physical world” [03].

This notion of ‘touchless touch’ was evident with Ultrahaptics, which also rely on action or gesture (see next section) where there is no physical contact with an object, and real-time interaction is achieved through gesture, such as a rotational gesture (simulating turning a knob) to change the volume of music.

Gesture-based interaction using immersive VR with Leap Motion and enhanced mobile applications was described by two other participants in interpersonal communication applications, that enhance: (a) creativity, fostering a sense of presence—a painting application “where people watching the artist could see the artists strokes and paint and could feel them too”; (b) connection, through contextually situating visual and haptic cues on two phones wirelessly linked that enable an illusion of communicative touch. Touching and moving a finger on the screen produces a light trail “it is like a magical fairy dust” on both phones. When the light trails cross from input on both phones, a vibration occurs at the cross point, eliciting a sensation of another person touching you; and (c) co-operation—through improved collaborative VR systems that enable people miles apart to interact in a shared virtual.

Through these applications, touch is perceived to foster meaningful social dimensions to interaction. Key aspects related to digital touch emerge: digital touch is connected to a sense of (co)presence and participation; the significance of context in enabling effective communicative experiences, despite the low fidelity of haptic sensation; and synchronicity or the timely implementation of haptics rather than its accuracy in terms of the tactile properties of the objects/bodies being touched.

4.5 Mid-air haptics

Two participants focused on mid-air touch or “touchless interaction” [07]. Given the novelty of this technology, work has focused on “what a mid-air touch experience can do, and … describing it”.

…you don't need any attachments … you need an array with ultrasound speakers. You … put your hand above and you feel single points. That feels like blowing in your hand, blowing through a straw so you can make it very focused or make it very dispersed depending on how you modulate the frequency range from, for example, 60–250 Hz [07].

In conjunction with neurophysiological research on mechano-receptors and hand touch sensitivity, the required size of the hardware used to generate ultrasound waves currently drives the focus of mid-air haptics research on the hand, rather than other parts of the body.

While this re-configuration of touch is driven by the way the technology functions, it demonstrates the potential of digital technology to create new ways of eliciting touch sensations linked to our everyday experiences. Here, the research focuses on the psychophysics or the perception of where users feel touch using mid-air haptics and the relationship to the emotional affective aspect of touch [07] (e.g. Hajas et al. 2020). This participant highlighted the role of contextual information in moving from the perception of touch to the emotional meaning of touch, and the research direction looking for mappings between mid-air haptic stimuli and the emotional reactions, happy touch or a sad touch [07] (Obrist et al. 2015). However, it also points to the significance of the sociality of touch, as users draw on these to make sense of and fill the felt gaps in VR touch experiences. Mid-air haptics is also an interesting form of VR in their inherent focus on the tactile (more than the visual) and their ability to create a touch illusion, that may have specific relevance to the visually impaired.

5 Summary

Technological capacity and functionality shape the kinds of questions being asked about touch, the body location as targets for the design and development of virtual tactile experiences, the implementation of the tactile and ultimately how touch is conceptualised and discussed across a broad notion of VR contexts. Questions focus on understanding which touch properties are important in specific contexts (e.g. training, rehabilitation), and the physiological sensitivity of parts of the body in generating specific tactile experiences (e.g. phalanges, palm of the hand). This focus leads to an emphasis on implementation of particular touch properties that are seen as critical to specific contexts, together with positioning and tracking—important for integrating visual with manipulative action. The implementation of this digital touch gives rise to specific ways that touch is conceptualised: how it is described, how it is realised (replication and illusion) and how the body is conceptualised, fragmented and brought into the interaction.

6 Discussion: conceptualising touch in VR

How touch is conceptualised matters in shaping both the landscape of touch in VR and technical development and design. How touch is conceptualised brings opportunities and challenges for design and raises questions for how touch is reconfigured and related to social implications. It shapes who touches, what is touched and how it is touched. It sets an agenda for the types of touch that are facilitated by VR, and whose touch is seen to matter. In this section, we discuss participants’ descriptions of touch, how touch is talked about as replication and as illusion, and how the body is brought into this ‘touch’ space.

6.1 Describing touch

Participants recognised the complexity of touch as a sense, together with the challenges in defining touch:

…there are all these other sense organs that are just overlapping and are tangled up. We call it "touch", but it is really a bunch of different senses [06].

Across disciplines and approaches within HCI, psychology, sociology and beyond, at a most basic level, we lack ways of describing touch itself (e.g. Classen 2005) and of talking about touch (Obrist, Seah and Subramanian 2013). We have only a ‘small vocabulary for speaking about touch’ (Finnegan 2014:198), the lack of a commonly agreed lexicon of touch and failure of the terms that we do have to describe the nuances of touch constitute ‘a very real hindrance to the progress in this area’ (Spence and Gallace 2013: 30), as experienced by one interviewee.

…how would we be able to tell developers and engineers how to build technologies if we don't know how to talk about it [07].

When talking about digital touch, participants used descriptions that situated digital touch relative to our physical touch experience, for example, a force pushing down links to a weight in our hands or kinaesthetic touch—manipulation and moving things around is part of the idea of touch [01]. Touch was talked about in physiological terms (e.g. vibration, texture, temperature and stiffness), as well as metaphorically (e.g. like silk, tickling, a spider or blowing through a straw on your hand). People made associations mapped to different parameters from the device to verbalise their experience. Participants also highlighted the importance of designing criteria to understand how people socially relate to touch [08].

A current lack of effective language or vocabulary to describe virtual touch or haptics was noted by several participants:

Currently there is no haptic language, no reference point for this. It is an emergent space in terms of building appropriate metaphors [06].

Indeed, technical language can develop across different spaces or disciplines, for example, HaptX uses the term ‘tactors’ for a form of tactile or haptic pixel, while others have used ‘haxel’ (hydraulically amplified electrostatic actuators) or ‘taxel’ (for electromagnetic pixels) (EPFL https://www.epfl.ch/labs/lmts/lmts-research/haptics/). Parisi (2020) highlights the cultural consequence of failing to develop a unifying language in relation to vibration, when the languages of touch are so intricately linked to the device or specific application.

That touch is hard to define, yet considered important for engagement and immersion, coupled with technical challenges, led participants (designers and developers) to conceptualise touch in two key ways: replication and illusion. These two interconnected conceptualisations of touch offer a starting point for thinking about social implications.

6.2 Touch as replication

Several participants primarily talked about touch in terms of neurophysiological dimensions. A biomimetic approach involving a fine-grained analysis of physiological touch mechanisms, grounded on how our body, skin in particular, perceives touch. From this perspective, the body is perceived to be like a machine consisting of parts with specialised functions. Touch is talked about in terms of information received by the body and processed by the respective sensors, depending on the locus and particular nature of the touch. Within sociology, this mechanical conceptualisation of ‘body as machine’ is critiqued at a macro-level for recreating particular types of productive and social bodies (Shilling 2008). The metaphor is seen as dehumanising, splitting mind, spirit and body, and resulting in a functional view of touch (Classen 2005). This approach contributes to the identification of physiological differentiation of touch (i.e. pressure, temperature, etc.), and inherently to the notion of ‘fragmenting the body’ (Sect. 5.4). Within this perspective, digital touch was couched in terms of replicating or ‘hacking’ the body mechanism to trigger sensations for the user.

The following quote illustrates how neurophysiological aspects are viewed as critical to the development of haptic technologies:

The sensory apparatus that we have underneath our skin… they're very small pressure sensitive, temperature sensitive systems. These are all firing signals up to the brain, telling us the different qualities of the objects we are touching. So the name of the game is to try and get things actuated in a similar way [02].

This conceptualisation recognises the sociality of touch in restrictive ways, neglecting the different acuities, sensitivities and preferences shaped by individual biographies, histories or cultures of touch, which influence how people come to touch in VR.

Research has shown how humans can distinguish between different types of touch and function well within specific stimuli ranges [07] and this has provided a foundational ‘frequency range’ for touch [06] for some participants’ haptic feedback research and development. These mechanisms of touch were central to informing the design of location and type of haptic feedback, while research on body-based perception of touch indicated ways in which humans perceive touch stimuli, and the ranges within which we optimally function. Again, the focus is on a ‘universal’ notion of touch sensitivities without unpacking this to engage with prior experience, culture and the sociality of touch in understanding what is ‘optimal’ and ‘normal’.

There are 8 receptors that we know today, doing completely different jobs and we think of touch as a single unified sense, but sharp objects get sensed by different receptors. I’m more interested in that part…. for us its always been touch but in dissecting it into different parts, what gives that particular feeling, the nice warm feeling of touching a soft surface? [09].

The physiology of touch, how our touch senses work and exactly what they are is still sketchy. While contemporary neuroscience claims that there are many more senses within the term touch, sensory anthropologists focus on how culture shapes the bodily notion of senses. A combined map of touch from such different perspectives could offer a more comprehensive framework for informing digital touch design.

To inform design, specific body parts and their physiology are considered in relation to how haptics can effectively and meaningfully mediate touch, for example, in mid-air haptics:

We try to map out where you can perceive it. It’s not very easy to perceive it on the hairy part, so it’s really on the non-hairy part, the glabrous part of the hand where you perceive it [07].

Experimentation with other body parts brings the focus of touch to ‘where air vibrations can be felt’ most effectively on the body. For example, stimulating the lips was found to be ‘very nice’, but less so for the ears, so “we focused on the hand” [07].

Understanding the perceptual qualities of touch went beyond the physiological for one participant whose focus was also on the “qualitative, so we really try to understand the subjective experiences as well” [07] which was seen as critical in the context of touch and emotion. A key part of designing technology that ‘hacks’ the body mechanism was considering ‘what is necessary’ to elicit desired sensations. For example, while acknowledging that there are multiple touch receptors, one participant highlighted four as key receptors (vibration, texture, temperature and stiffness), suggesting that if four characteristics allow the classification of 92% of the human touch, then there is no need (and it is cost effective) to disregard others [05].

The implications of this analytical dissection of the phenomenon for the identification and naming of different functions and types of touch are twofold. First, it demonstrates the importance of neurophysiological research for haptic technology development and design of optimal, effective touch experiences that relate to the capabilities and limitations of the sensory body. Second, it brings awareness to the design benefits of a more holistic approach, which takes into account that the body is shaped by the sociality of being in which perception of touch is about neurons and cutaneous receptors, and is also embedded in the social, cultural and historical aspects of our touch experiences.

One participant highlighted some key issues with touch ‘replication’, wanting to move beyond notions of ‘realism’ to think about touch differently in VR, akin to cinematic features. (This quote links back to the issue of lack of standardisation of ‘haptic language’, as well as creator community tools and playback devices):

If you think about Hollywood movies, they are cinematic, they are real at some level but they also rely a lot on what we call film language. We all have seen enough films that we know that when this happens that is what the director means to tell us. Even documentaries are not realistic. So this idea that we need to replicate touch reality is a huge distraction that I am constantly fighting against. It is hard because there is no film language for haptics yet, there is no haptic language and there is not that kind of widespread creator community that is fostering that, there is really no reference point for it [06].

In short, while neurophysiological knowledge, physiological factors and perceptual reproduction in terms of interpretation of haptic sensations are critical in understanding key touch mechanisms, a sole focus on the nervous system and the physicality of the tactile experience do not provide convincing answers about the social dimensions of touch (Classen 2005, 2012). How we make sense of touch, how we receive, interpret and respond to touch are intimately connected to and shaped by the social, cultural and historical aspects of our touch experiences. Deconstruction of touch into subcomponents raises challenges for how people interpret deconstructed touch sensations, for example, pressure as weight or pressure as soothing touch (Price et al. in prep). In terms of design, this raises questions about which facets of touch properties to foreground or augment in VR experiences, which may be a function of which facets users can easily compensate for through social imaginations, prior experience and expectations.

6.3 Touch as illusion

Touch illusion in the context of sensory perception has been extensively researched (e.g. Massachusetts Institute of Technology 2008) and taken up in design within VR and other contexts, for example, conveying online textile quality (Orzechowski et al 2011). Several participant conversations around touch also emphasised this ‘perceptual’ role in interaction, which is central to conceptualising ‘touch as illusion’. This concept aligns with a VR epistemology focusing on a sense of immersion where ‘illusion’ is seen as significant. Participants talked about imagination, mind tricks and perceptual gaps to bring achievable ways of mitigating technical and physiological challenges.

We're trying to get people to believe in something that is not actually there [02].

These participants talked about touch in terms of perceptual processes rather than dimensions of physicality or physiology. The emphasis was on understanding how the brain works in order to either take advantage of perceptual gaps or identify how the human brain infers the sensation of touch to exploit it for illusory effect.

A large element of the [VR] system is the person, their brain, their perceptual system and a large part of the game is in fooling that …. We are used to thinking if we feel this force pushing down here then we have a weight in our hands [02].

Materiality was generally considered a key component in creating touch illusions. Tactility—a sense of the material—is conveyed through haptics, which provides a cue that you have touched something. Modal synchronicity (signalling the importance of latency) is central to achieving these illusions of touch:

You have your haptic point and your visual point and they should be together. And sometimes the haptic point goes there (he points into the table a bit lower from the surface) and if you keep the visual there the people will believe that [02].

In ‘touchless touch’ contexts, this materiality changes, as the aspect of physical sensation is removed. While binaural sound has been shown to bring about a feeling of touch and even heat (Mead 2003), movement and gesture in conjunction with the visual and/aural also create the illusion of touch, shaping touch actions and perceptions. This critical role of other senses in achieving touch illusion is central to VR experiences, in particular, the interplay between vision, audio and/or haptic feedback (e.g. Zhang et al. 2019). Modal integration was explicitly raised in relation to applications that used visual and sound effects to generate a sense of touch without the use of haptic sensations. For example, in the rehabilitation context, the sense of materiality is present in relation to the virtual object, which shapes the gesture and action in manipulating the object and the perceived tactile sensation generated [03]. The visual representation of the object in VR is central to guiding the desired movement, “because the immersion in VR tricks the brain into believing that the object is real” [03]. This leads to new mechanisms for ‘touching’ or ‘holding’ for users who do not have the strength to hold a physical object (see Fig. 1).

Fig. 1
figure 1

Holding a controller (right hand), holding and balancing a bowl

Full size image

Similarly, with Ultrahaptics, the materiality of touch is challenged and notions of touch are folded into action and gesture. Technology connects the action or gesture that might typically accompany touch interaction but without the object (physical or visual) itself (e.g. a car radio volume knob), precluding the need to be physically in contact with something (e.g. the radio button in your car). This type of ‘touch’ interaction raises questions concerning the shaping or development of gesture. Without the materiality of the object to constrain the gesture, for example, do touchless gestures become less accurate; what are the design implications of this; “Do we feel sense of agency—the sense that we are controlling something, if we are not touching it?” [04].

The electrifying cloud immersive experience (Tonandi Experience by Magic Leap), mentioned by two participants and noted earlier in the paper, comprises a visualisation of an ‘electrified’ cloud accompanied by a ‘crackling’ sound and sharp buzz, which is only made when the user touches the cloud, inducing users to associate it with an electric shock.

figure a

Magic leap electrifying cloud from tonandi experience: https://www.youtube.com/watch?v=5iTLN3AuBws

There is no haptics obviously but it sounds and looks like you are getting shocked of electricity and it felt like that to me. I actually had a phantom hallucination that I was touching something [06].

This example highlights the centrality of modal synchronicity. These digital forms of touch allow for engineering touch that is not sensorially possible in the physical world: the touch of breath particles, or the touch of a flower.

The idea was to simulate the feeling of touch through vibrating your wrist when you waft your breath–the particles just going around–that moment, that touch [09].

These examples illustrate that materiality is brought differently into question in virtual spaces, adding complementary experiences of touch, which do not require direct mapping to experienced physical phenomena. In particular, it brings awareness to the notion of ‘near touching’ or a sense of presence we might feel, for example where you might feel (touched by) someone running by you. While this reconfiguring of touch is to some degree a function of technological capacity, it nevertheless brings new touch experiences and awareness and shifts the boundaries around what we might define as touch, expanding the potential for digital touch-based interaction experiences (e.g. touching the invisible).

A number of challenges emerged in relation to conceptualising touch as illusion in VR. There are reduced touch sensations, for example, temperature is not typically included, and prolonged touch is problematic. Illusion is also grounded on people’s expectations, thus comes with a ‘locus of believability’, i.e. if the boundaries are pushed too far (e.g. a person can lift a rock but not a boulder), then the illusion breaks, foregrounding the need for sensations to be ‘accurate enough’, which returns us to the limitations of current haptics. However, future work could explore whether there are any benefits to audiences in being able to break the illusion of touch.

6.4 Body fragmentation

Haptics will remain a challenge since we sense touch all over the body and having systems that enable that is very complex [02].

When the body was brought into discussions around touch, participants focused on specific body parts, almost exclusively on the hands with some reference to the lips, face [07] and feet [09], and in relation to technological design, affordances and limitations. While the fragmentation of touch is continuous with the origins of touch science as it developed in the 19th century, and with new psychophysical research on touch within the field of ‘haptics’, four critical factors emerged from the interviews that highlight this fragmentation and deconstruction of the body:

Firstly, the affordances and limitations of specific technologies determine where touch is experienced on the body. Current VR technologies clearly focus on the hands as a locus of touch through controllers, gloves, mid-air haptics and even gestural-based interaction. This is perhaps not surprising given the dexterity and central use of our hands in everyday interactions in the world. However, when we are touched, the location of touch on our bodies is much more varied, e.g. a touch on the shoulder in sympathy or reassurance, stroking a friend’s arm, stroking a face, giving and receiving a hug, touching your body against another’s. Thus, in terms of social touch communication in VR, current set-ups are somewhat impoverished. The touch afforded is more object orientated, professional and work-related, rather than supporting familiar, comforting, close or intimate touch. In this way, we relate the design of ‘virtual touch’ for doing things to objects or being ‘productive with ones hands’ to the sociological critique of bodies as machines. This perspective implies touch is for doing, not for being or feeling with others, which intersects with the design of touch as replication within VR.

Secondly, design of where digital touch is experienced is related to physiological sensitivity. Neurophysiological analysis fragments the body, through the focus on specific mechanisms involved in generating touch sensations, for example, C-afferents, mechano-receptors, phalanges, palm, upper limp, hairy vs non-hairy parts. This leads to the investigation into an isolated aspect of touch (e.g. how to get a specific sensation) to generalise to broader sensory experiences (e.g. holding an object) that are context bound. This process of re-couching touch as a reduced sensory experience that happens in isolation rather than as part of an embodied sensory presence in space and time may lead to a form of standardisation, or ‘one size fits all’ touch experiences. This universalisation of touch erroneously assumes that people sense things the same way or more problematically (further) governs a normative sensorium. Ultimately, this may reduce possibilities of exploring other modes of touching and curtail the freedom of tactile interaction as experienced in the physical world, bringing regulated, inflexible sensory experiences into VR contexts. This sensorial reductionism aligns with the metaphor of ‘body as machine’ a form of analysis that breaks down actions into a series of constituent elements, eliminating all inessential movements to produce “movement that was precise, regular and inflexible” (Classen 2005 p.168). This functional approach to touch might be seen as dehumanising, as it fails to engage sufficiently with the social, cultural and historical aspects of our touch experiences.

Thirdly, the body is fragmented through considerations of what properties of touch to convey (e.g. weight) or the kind of tactile sensation that will provide meaningful and critical information. The isolation of specific touch properties foregrounds particular touch sensations in digitally mediated experiences and leads to their deconstruction. For example, with haptic feedback, you might feel the presence of an object in your hand, but not its temperature or texture. This deconstruction of touch generates a new ‘stripped back’ reduced form of touch: deprived of some touch properties that would ordinarily be sensorially experienced when touching a physical object. The reduction or removal of extraneous information brings a focus to the sensory aspects being foregrounded, while raising the question of who decides what is ‘important’ (the designer or the user). This instigates an interesting tension within VR between touch as replication and touch as illusion, each placing different considerations on what is ‘important’, but each leading to different reductions and choices made in relation to touch. Illusion is grounded on ‘what is sufficient’ to effectively feel what you are touching in VR. In contrast, replication focuses on designing ways to mimic neurophysiological responses. Both approaches result in impoverished partial forms of ‘experienced’ physical touch. While historically haptics has resulted in partial forms of experienced touch (e.g. Atkinson et al. 1977), ‘framing’ touch as replication through a focus on neurophysiological sensations might also exclude attention to cultural and social meanings of touch interactions, which are less evident when conceptualising touch as illusion. Nevertheless, these findings raise questions around reproduction and modification—or designing to replicate touch versus re-imagining touch beyond its instantiation in the physical world. Current technological affordances can also serve as resources for inspiration. For example, touch instantiated differently through developing devices, even in a deprived, abstract or even ‘symbolic’ way widens the possibilities of what touch is or can be in that space with the potential to move towards new (as in ‘different’) touch experiences.

Fourth and finally, consideration of where a tactile sensation is best received on the body for its intended purpose contributes to body fragmentation. While technically specific parts of the body are integrated into the feedback loop, and function as ‘nodes of connectivity’ between the technology and the physical body (Cranny-Francis 2008), this location fragmentation has social and design implications. In particular, ‘body accessibility’–or our willingness to let others touch our body (Jourard 1966)–is of critical consideration. The hands, head and arms have been shown to be the most ‘accessible’ parts of the body to touch in Western cultures, although work on digital touch imaginaries suggests that “bringing touch in the wider body, even if moving beyond the hand, up the arm, across the shoulder, raised participants’ concerns about the appropriateness and control of touch” (Jewitt et al. 2020, p.60). Women have been shown to be more discriminating about where they are touched, and men are more concerned with the kind of touch than its location (Moore et al. 2014). “Consideration of the social norms of touch is therefore significant for the use and design of digital touch—whether attempting to work with, against or to reconfigure them” (Jewitt et al. 2020, p.59).

7 Conclusion

We have evidenced the value and growing significance participants placed on touch in virtual spaces. Collectively this shaped how touch is talked about, conceptualised and implemented in VR, its development and design, and consequently the ways we can and will experience touch in VR.

Across the interviews, touch was linked to a notion of connection and seen as a way to bring a ‘sense of realness and presence’ in VR (e.g. Paterson 2006), yet noting the critical role of the visual and aural in achieving this sense of ‘real’. While notions of replication and illusion both speak to ideas of ‘realism’—they are both seeking to reach some form of ‘realistic’ touch experiences—the partiality of touch emerges as central across the themes discussed above and is also representative of the complexity of describing virtual touch and the challenges in neatly defining it.

In conceptualising touch as illusion technology does not provide like for like touch sensations, yet still aims to create a ‘believable’ sense of touch. This touch is partial in having less of a physical dimension (in the sense of a degraded haptic sensation), and reliant on sensory cues. In particular, touch memory (including movement and gesture), precise synchronisation of other sensory representations with action, and context and emotion are brought into touch perception and interpretation. Illusion is grounded on ‘what is sufficient’ to effectively feel what you are touching in VR. In contrast, conceptualising touch as replication focuses on designing ways to mimic neurophysiological responses. Thus, the conceptualisation of touch strongly informs the design process. Concepts of replication and illusion both speak of touch in relation to a ‘fragmented’ body. Body parts and tactile sensations come into play as isolated elements rather than part of a whole (body). We argue that this deconstruction and fragmentation are part of a technologically driven investigation, which focuses on how to bring the body into the virtual world so that it can experience touch. This brings challenges for design (identifying key sensory experiences in a situation, and what is ‘sufficient’ to experience ‘touch’), and the social implications of how touch is reconfigured in the VR landscape.

The renegotiation of touch emerges as a central debate. If the tactile is isolated from all other cues, touch sensations are relatively uniform (i.e. a vibration) and partial. This has implications for the meaning placed on touch interaction, and calls for the need to attend to social norms in our touch experiences—given that different interpretations may result from specific touch sensations (e.g. gender-related nuances). However, such limitations may offer more ‘focused’ touch experiences considered valuable in certain contexts (e.g. medical training). Focusing our touch sensations by isolating touch elements or determining where touch will be felt gives rise to engineering ‘new’ emphases on touch experiences, which foreground specific characteristics or deprive others sensations. This has two key implications. Firstly, it fosters an efficiency led design approach to touch. The limitations and challenges of digital touch were acknowledged by all participants—most focusing on hardware and infrastructure. These challenges demonstrate that digital touch is realised in a confined, specific space and shaped by values (Welchel 1986) related to technical capability, physiological knowledge, cost-effectiveness and context specifications—based on what is determined to be ‘needed’ in specific contexts. Secondly, it raises questions around power–who is determining the reconsiderations, that while guided by technological affordances and limitations, and the task and context to shape the meaningfulness of the intended touch, also need to take into account ethical issues.

We have argued that how touch is conceptualised shapes how technology is seen as a new space or locus for novel touch experiences. Firstly, in the sense of being different from physical touch, new haptic experiences can be generated in relation to tactile engagement. Secondly, meaning generation around these novel haptic experiences occurs as an interaction between users and designers bringing new ways of talking about touch. This links to a third point that while replication and illusion hinged on realist aspirations these technological techniques can also be employed towards interpretive designs of touch. Looking beyond realism as a target for design allows for the integration of creative artistic approaches to touch design, framed by the affordances of technology. This allows for new ways to look at touch and haptics. The generation of a haptic language might also foster new ways of thinking about virtual digital touch beyond realism. Fourthly, materiality changes in VR environments. The features and behaviours of objects related to the impact of touch (e.g. fragility, plasticity, decay, destruction, death) are programmable in novel ways, for example, if something fragile no longer breaks you can squeeze it, stretch it, throw it. As a result of this reconfiguration and virtual materiality, the types and norms of touch in the virtual world can differ from those of the physical world. Non-realistic tactile and material engagement can distort and disrupt touch in ways that have implications for disrupting established ‘real world’ socialities of touch as well as their renegotiation by users in the space of digitally mediated touch in VR where the boundaries between touch in the virtual and the physical world are blurred and in flux.

This paper contributes to our understanding of how the conceptualisations of touch in VR raise socially orientated questions for design. However, further work needs to engage with touch in relation to other contextual elements in VR. The social implications of touch in multiuser contexts versus single user contexts (the focus of participant discussions in this paper) are considerable, given the cultural norms that sit around who and how we can touch and be touched by. We also need to better understand how the different types of devices or means for bringing about touch can best be mapped to or used in relation to different types of content, as well as touch experiences with devices or physical artefacts integrated into the VR experience (e.g. a physical object representing a tree, or a physical weighted sword in ‘Dragon Quest VR’) in a more mixed reality space.