1 Introduction

After a long gestation, Virtual Reality (VR) is finally the customer-ready tool that fulfils the dream that scientists and fictional novels’ authors had back in the ‘80, when VR was first theorized. However, despite its astonishing visual and interactive capabilities, there is an element that still misses, compared to what VR was imagined to be: realistic haptic feedback. While most of the commercial headsets are shipped with standard controllers that provide a more than satisfying experience, these devices are designed to be as close as possible to console gamepads to avoid a sense-shock [20], and are capable of providing only a limited set of haptic feedbacks (e.g. controller vibration). Almost a decade after the introduction of VR as a customer-ready product, haptics is still far behind, and nothing seems to suggest that it will be able to catch up any time soon. While multiple technical reasons can and must be addressed, in this paper we focus on users’ perception of haptics and their attitude towards possible usage of such devices. The history of gaming - and virtual environments in general - is dotted with countless attempts to make haptic devices appealing to the market, none of which has ever managed to reach any financial sustainability nor has been generally acclaimed to be a technical breakthrough that would justify a significant interest by the general public.

To increase their appeal, haptic devices may try to follow the VR path, building peripherals that can be used in a wider range of contexts and are not bound to a single scope that derives from mimicking a natural behaviour. The idea behind this preliminary study is that, to become more appealing, haptic devices should be built to support many different interactions, either natural or metaphorical, so that designers could find the best implementation for their experience, some of which unthinkable by the hardware manufacturer themselves. Following this line, we defined the multimodal haptic interaction as the co-presence, in the same VR experience, of both a natural interaction and one or more metaphorical interactions (not necessarily connected with an actual physical interaction with objects), applied by a single haptic device. In this paper we describe the preliminary results of a study aimed at understanding the impact of multimodal haptics on audience’s interest, arguing that haptic devices that are built with a single function in mind can be viewed as too narrow in their scope to appeal to a broader audience, which fails to see other possible uses beyond the primary one. Our intuition is that building haptic devices with a single purpose, or that mimic only one natural behaviour, contrasts with current scopes of VR, which is arguably one of the most versatile systems of this era in terms of diversification of experience. Also, a secondary objective of this work is understanding the audience’s reaction to a combination of different haptic stimuli, in particular, if metaphorical interactions inserted in a context that continuously provide natural interactions, can generate confusion or discomfort.

To test our argument, we created an immersive virtual environment that had to be played with a wrist exoskeleton and tested it with a crowd of experienced haptics scholars. This choice was motivated by two factors: the very low degree of familiarity that even an expert haptic user, such as the testers we gathered for our experiments, may have with wrist exoskeletons, and by the fact that focusing on a single joint could help us design more precise and robust feedback for both natural and metaphorical interactions. Considering, in a holistic meaning, multimodal feedbacks, graphic stimuli played a complementary role with haptics, helping the player to feel the immersive experience. Results suggest that by designing a gamified experience that used the exoskeleton in many different ways, we were able to stimulate a strong interest towards it, and therefore make it more appealing.

2 State of the art

Virtual reality (VR) systems have always received great expectations, as a big new step towards high-realism experiences, whose meaning involves both immersion and presence as the main blocks. Whatever videogame, military, cultural heritage, phobia treatment experience may be in analysis, the key factor to successfully bring a satisfying experience is to enhance the subject’s sense of “being there” [2, 6].

Several previous studies empirically demonstrated the benefits of an immersive system, against a non-immersive one [13, 19] e.g. a WIMP-based (windows, icons, menus, pointers) system.

In the last two decades, plenty of attempts have been made by universities and manufacturers to create low-cost, ergonomic and appealing products. Nowadays, after the involvement of big hardware manifacturers, three main bbrands dominate the market: Oculus, Valve and HTC. Each VR system producer designed its system following a specific principle, building headset and controllers with pros and cons on the overall playability performance, but all keeping the direction of gradually replacing the old joypad-based systems with a finger-tracking system. Those systems go towards the paradigm of a so-called natural interaction, bringing the technology to something people are used to interacting with in everyday life, e.g. through perception, gestures, expressions and movements [23]. The naturalness paradigm allows computer systems to exploit the perceptive abilities, the main feature of human-human communication, also improving the learning capabilities [7].

On the other hand, keeping the hardware fixed, VR systems can still count on metaphors or metaphorical interactions, that are, just like in language, a surrogate in the service of the real meaning of something, in this case either an action or gesture. They try to cross the frontiers of everyday life experience and expand the human capabilities, transferring concepts already known by the user on a certain context on a new task in a different context [8].

As a general effect, metaphors facilitate the dialog between the user and the environment and improve the intuitiveness of the interaction, when provided by devices that are not too shocking for their habits [24]. Examples can be a circular hand motion to stir a pot or a thumb up or down to communicate with an NPC.

To achieve a charming and astonishing experience, all the human senses should be stimulated in a virtual environment, exploiting visual stereoscopy and spatial sounds, encouraging the natural interaction through the use of haptics, which provides the user with force, tactile or proprioceptive feedbacks [5].

Current state-of-the-art haptic devices are used in a predetermined scenario in order to accomplish a certain kind of perception task, coupling a single device with a single type of feedback, depending on either the application, mechanical properties or the stimuli type [18]. Since the human interactions with the environment are exerted through limbs, almost all the haptic devices in previous studies aimed at transferring this interaction either in teleoperation systems or virtual environments [14]. While just a few works investigated the effects and usefulness of haptics during virtual walking [21, 22], a lot more focused on perception tasks on the upper limb, hand or fingers only, using fingertips, exoskeletons and many other custom devices. Many works in the state of art demonstrated how tactile feedback can be given using fingertips to either perceive objects shape or roughness [11], improve the grasping capabilities in physics-enabled environments [16] or sensing objects’ temperature [10]. Instead, exoskeletons have been used in different scenarios for receiving force feedback within teleoperation systems [4, 15], with several control strategies (e.g impedance control).

In all those studies, haptics renders a single physical perception task, supporting the aforementioned natural interaction paradigm. Together with visual and auditory information, haptics enhances the sense of presence in a virtual environment without, necessarily, improving user performance in the observed task [17]. Nevertheless, from a technological point of view, haptics still cannot bring into a virtual environment a complete set of perceptive natural interactions that would fulfil the sense of presence of the user, slowing the diffusion process into homes [3].

Starting from this limitation, it comes the idea of enchancing the natural interactions of haptics-ready environments with metaphors. While natural interactions anchor in solid physics principles, metaphors depend on the developer creativity only, still respecting two important constraints: fitting the user’s previous knowledge and fitting the physical constraints on the interface [25].

To the best of our knowledge, an attempt of natural and metaphorical interactions fusion, in the same VR experience, still misses, as well as an analysis on how users perceive metaphorical interactions in a haptics-ready VR environment, provided by a single device. Multimodal haptics, thus, has the aim of bridging the current haptics interactions gap and, concurrently, making the user experience new abilities in the metaphors space.

3 The island

To gather data for the experiment, a small VR game called The Island was developed. This game was showcased during the 2019 WorldHaptics conference in Tokyo, Japan, as part of the 2019 WorldHaptics Student Innovation Challenge. Conference attendees could freely try the game and provide feedback, either in free-form or by filling out a short questionnaire, at any time during the event. Due to time constraints, not all attendees were able to fill the questionnaire after the gameplay, because of conflicts with their schedule.

3.1 Setup

The experiment ran in a clear 3x3 meters area tracked by an HTC Vive VR system. Users were instructed by an operator on how to wear the headset and the Exiii wrist exoskeleton around which the game was built (see Fig. 1).

The HTC Vive VR system (HTC, 2020) is a new-generation headset that provides two infrared tracked controllers, localized by two infrared laser emitter units through a so-called inside-out principle, where the time difference at which the headset photodiodes are hit by laser rays emitted by the two emitter units (called Lighthouses) is computed, giving position and orientation of the headset. The controllers were used to allow exploration (with clamped low movement speed to avoid motion sickness) and grab/release actions of highlighted objects. Users were able to freely explore the environment (within the experimental area boundaries), both by walking within the aforementioned area and moving the controllers.

As it regards the haptics, a wrist exoskeleton namely Exiii Exos DK2 (Exiii, 2020) was used as interface. The Exos is a light 2 DoF wrist exoskeleton that provides counter torques (max 20 N*cm) in palmar/dorsal flexion and radial/ulnar abduction. It also features a stable wireless connection to the host PC, avoiding cable obstacles.

3.2 Software

The Island is a game-like VR experience developed using Unity. It is composed of four different scenarios, each one with its unique sets of features and a unique usage of the wrist feedback. Furthermore, as a basic working principle, a force-feedback is given to the user every time a collision with a virtual object occurs, featuring a ground level of natural interactions. The game has been designed as an immersive experience with a sequence of events that provide a sense of progression, with the user that starts on the island shores and ends up finding the treasure room with the loot. The rationale behind these designed games relies on providing the player a context that is very common in fantasy stories and cartoons so the player can expect ’weird phenomena’ to happen. Also, the features of each level have been conceived to deliver the aforementioned two types of haptic interactions. The sequence of levels goes as follows:

A. Labyrinth: It is the first level of the game. The player wakes up on an island. The environment appears as a sandy beach with a maze surrounded by forest (block “A” in Fig. 1); This level implements 3D spatial sound, with the sea waves effect fading away as soon as players walk away from the sea, approaching the maze. The aim in this level is to find the maze exit, exploring all the paths within. Six invisible checkpoints are placed on the ideal exit path, giving an auditory sound when hitting them. In this scenario two types of haptic feedbacks were provided: a metaphorical one linked to the navigation of the maze and a natural one provided when touching objects in the environment (e.g. walls of the maze). As soon as the player enters the maze, a metaphorical continuous force is given on the player’s wrist flexion/extension DoF, guiding him/her towards the exit. The amplitude of this force is proportional to the angle between the vector that links the player position to the checkpoint one and the controller direction vector. Thus, the more the player is going far away from the right direction, the bigger is the applied force to the wrist. Since no perception actions on nearby objects are experienced by the user in the same moment, the metaphor just attaches an additional information layer, enhancing the already existing natural interaction.

B. Tribal threat: after exiting the maze, the player reaches a big circular open space, delimited by dense forest (block “B” in Fig. 1). In the middle of this open space lies a human skeleton, with a shield on top. To forward into the game, the player has to grab the shield. Once it happens, a few arrows start to rain towards him from the jungle, getting denser and denser by the seconds. Every time an arrow hits the shield, the player experiences a natural interaction, that is a set of torques on the 2 actuated DoF, computing the distance between the shield centre (i.e. the player wrist position) and the collision position between the arrow and the shield itself. After succeeding in parrying all the arrows, the player can reach the entrance of a cave in front of him. No feedbacks are provided if the player does not parry any arrows.

C. Blind path: at the beginning of the cave, the player finds a torchlit dark environment and a wooden stick leaning on the sidewall (block “C” in Fig. 1). In front of them, a long black corridor that cannot be walked, as it is too dark. If the player puts a foot in the wrong position, he would jump into a pit, and start the level from scratch. To reach the other side of the cave, the player has to grab the wooden stick on the wall and use it to perceive the ground, feeling the floor reaction force on the wrist and, thus, avoiding all the empty zones. This interaction is the most natural interaction possible. The player has to grab a tool and use it, just as he would in reality.

D. Treasure: at the end of the dark path, one last trail keeps the player away from the treasure. On this path, wooden logs and thick spider webs block the road (block “D” in Fig. 1). In this level, the user had to grab a sword which they could find near the path, and grab it by getting its hand closer. Once the sword was grabbed, they had to hit all the obstacles on their path. The haptic device returned a different force feedback based on the object, with a dedicated feeling received every time the sword penetrated with an obstacle. After this, the player has to smash a big wooden door, to finally collect the treasures behind it. This natural interaction is similar to the one in the previous scenario, but this time players could feel the different feedbacks that were provided by the same movement, adding a level of immersion to the whole simulation. Just as one would expect to be, minor and imperceptible feedback is provided while cutting a web, while relatively strong and hard are provided while hitting the door.

3.3 Experimental protocol

The experiment was designed to exploit all the VR and haptics features, through all the levels, without particularly focusing on either natural or metaphorical interactions: the aim was merging the two interactions in the same VR experience, without pointing out the difference between them. As already previously pointed, the exoskeleton was used not only for perceiving the environment objects (natural interactions), touching them, but also for both receiving indirect (through a grasped virtual object) force feedbacks (e.g. Tribal Threat, Blind Path, Treasure) and gathering additional information to complete the game (i.e. metaphorical interaction in Labyrinth). In the unlikely event of a concurrent natural and metaphorical interaction, the software prevents it only providing the natural one, since users were spontaneously more focused on the physical interaction during a touching action. Visual and auditory feedbacks were provided as game clues to be complementary with haptic ones. Also, in the case of natural interactions, a redundant visual feedback was provided together with the haptic one: whenever an object was touched, an additional user hand skeleton was drawn overlaying the existing one, and the further the former was than the latter, the higher was the force provided on the wrist.

The authors’ insight is that in complex environments the haptics rendering could be useful for achieving multiple metaphors in addition to the classic feedback set without altering the natural interaction haptics is supposed to provide.

First the subject entered the experimental area and wore the headset, adjusting the strap band around the head. Then the wrist exoskeleton was put on the dominant hand, tightening the strap bands enough to reduce the mechanical slack of the interface, together with the controllers (see Fig. 1). Each subject played a minimum number of two different levels. Virtual walls appeared in the virtual environment whenever the subject position was close to the area boundaries, avoiding unexpected and potentially harmful collisions. At the end of the game experience, subjects were asked to fill a questionnaire, answering only to the played-levels related questions.

Fig. 1
figure 1

All game levels on the left (from top, labyrinth, tribal threat, dark path and treasure) and experimental setup on the right. Detail of experimental system architecture on top-right

Full size image

3.4 Questionnaire

At the end of each playthrough, players who had enough time were asked to fill a 12-items questionnaire, specifying which levels they played. Each question asked the subject to rate the answer on a 7-points Likert scale, ranging from “Absolutely Not” to “Absolutely Yes”. Questions were related to previous experience with VR systems and haptics, consistency of senses, information perceived in the environment, haptics usefulness and realism, labyrinth navigation, ground force reactions realism, versatility of the haptic device and playability Table 1.

4 Results

Out of 23 people who tried the game, only 12 of them provided feedbacks, 9 of which used the questionnaire, while the other 3 were members of the committee who evaluated the game as part of the challenge. Randomly choosing the levels for each subject, led to a mean number of 5.5 subjects per level.

Starting from users’ profiles, they all reported a high degree of familiarity with VR (Mean 5.8, SD 1.16) and haptic devices for VR (Mean 5.6, SD 2.06), and the majority of them have already tried something similar in VR as well (Mean 5.7, SD 1.30). All participants believed that haptic devices could potentially be useful to improve immersion and presence in virtual environments, answering 6 or above (Mean 6.55, SD 0.52) to the question, giving similar answers when asked if they believed that the haptic they just tried could be useful in other scenarios (Mean 5.6, SD 1.00). For what concerns users’ appreciation of the experience, players were enthusiastic and they all expressed potential interest to play again (Mean 6.5, SD 0.72), with six testers out of nine answering 7. They also believed that the usage of a haptic device increased realism of interaction (Mean 5.8, SD 1.26), and that information received by other senses was coherent with the interaction (Mean 5.7, SD 1.09). Users also believed that the haptic feedback was helpful to complete the tasks they were faced with (Mean 6, SD 1.00), showing a particular enthusiasm on the implementation of the labyrinth’s interaction (Mean 6.25, SD 0.88).

As it regards the freeform written feedback, reviewers seemed to appreciate the navigation, calling it “very intuitive”. However, they had mixed responses to the Tribal Threat level, with one reviewer focusing on the emotional impact it had (“It was very scary”), one liking the interaction, and one saying that arrows’ impact was not too precise. Similar feedback was provided for the dark walk, with a reviewer liking the idea “a lot”, suggesting that it would be an interesting tool to try in combination with applications for blind people, while another reviewer did not find the feedback realistic. The last level, the treasure room, was the least liked one, as the “swords hit on the wood was not realistic”. In addition, one reviewer concluded with “would love to play again, but on more complex games”.

5 Discussion

Despite the rather small number of testers, the app received overwhelmingly good feedback, with all subjects who would be happy - if not enthusiastic - to try the game again. This is unexpected, as the game was developed in a rather short time, with limited resources and no professional 3D modeler involved. Despite that, users appreciated the degree of novelty of the application, as well as the peculiar interaction device. As much as they were accustomed to haptics in combination with VR, wrist exoskeletons are very rare, and demonstrating that it can potentially be applied in a non-conventional way seemed to spark an interest towards the tool. Such a high interest can also be motivated by the extremely ludic nature of the experience: while most tech demos tend to focus on hardware’s features, the VR experience showcased the haptic device through proper, legit game, giving users the feeling of really using their wrist to control the world.

From the feedback point of view the multi-modality of haptic feedbacks showcased in this work was perceived as a high engaging feature of the scenario. This outcome came from the positive subjective rating of the labyrinth scenario and the absence of negative comments regarding the co-presence of metaphorical and natural interactions. Also auditory and visual feedbacks contributed to the multi-sensory coherence, given the non-redundancy of these signals. It is worth highlighting that the intrinsic non-physical nature of metaphorical interactions allows them to be designed in every possible way towards the need of either the application or the player. Thinking about non-trivial ways this can be beneficial to users, rehabilitation could be a potential field of interest especially in the context of reconstructing or reconceptualizing the effects of a trauma. Within this, metaphors already showed to be effective in such therapies [12]. Also in the same context, the effect of the co-presence of natural and metaphorical haptic feedbacks has never been investigated, to the best of the authors’ knowledge.

5.1 Interactions

As expected, the combination of multimodal haptics and VR has proven to be extremely effective. All levels have received positive reviews, with the labyrinth, the only one that made use of a non-behavioural interaction metaphor, to receive the highest degree of appreciation. Attendees who tried the application were unaware of its purposes, as no description of the experience was provided prior to the test, and they were surprised to feel that the exoskeleton was used in a completely non-natural, yet “reasonable” way.

The tribal warfare level has also received very high feedback, especially from the reviewers (it was not chosen very often as a random level during the demos). One of them especially reported a state of fear while playing it. Unfortunately, given the limitations of the Exiii development kit and the development time, the practical result was perceived as unrealistic by some.

While receiving mixed reviews, the third level, the one where people had to walk in a dark cave, is the one that received the most comments on. Despite not being designed to mimic the behaviour that disabled people may have, almost all users believed that the interaction was created with that goal in mind. In hindsight, we argue that, except for the last level, all the other three ones could all have been adapted for visually impaired people, being surprisingly versatile.

5.2 Potential biases

While the results seem to indicate an enthusiastic response towards the application, the rather small sample we collected is representative of a very narrow group of people, who are experienced with haptic devices. Given the point of this paper, this may sound as a point towards the aforementioned thesis; however, it is not possible to generalise the interest towards the haptic device to other groups of people who may have an interest in such devices (i.e. video game players).

Also, the game was showcased during a conference, as part of the final act of a student challenge where the whole point was to implement a wrist exoskeleton in VR in an intuitive way. It is possible that reviewers could be influenced by the order in which they tried the experiences, being induced to be more or less lenient towards our product after trying one of different value.

All these potential biases are a consequence of the testing ground of our first prototype, and further investigation will be performed when current limitations to user studies will be lifted.

5.3 Limitations

As already mentioned in the introductory chapter, there are other factors that may play a role in preventing haptic devices from reaching a broader audience. One potential obstacle that could be impacting the limited distribution of custom haptic devices is the cost. Current customer headsets are more accessible than in the past, but their price still ranges from three-hundred to a few thousand euros depending on the model. Haptic devices would be a significant addition in costs to an already expensive device.

Another potential issue to be considered is players’ position. While most VR experiences require the users to be standing in open spaces, there is still a significant amount of experiences that allow, if not encourage, players to be sitting. Having an additional device to be controlled in world space while sitting in front of a desk in VR could be an hazard.

Lastly, not all game genres could intuitively benefit from the addition of haptic devices. VR games are all designed around the visualization device (the headset), and in some cases around the specific haptic devices that are already shipped with it. Additional haptic devices may not fit all scenarios, making the additional hardware only usable in specific cases (similarly to what happens with steering wheels and flying sticks in common racing and flying games). Also our initial questionnaire did not cover any question in this regard, and further investigation will be required.

6 Conclusions

In this paper, we described a preliminary experiment aimed at exploring how to change the perception of haptic devices can change when multiple purposes are presented, with a particular focus on VR integration. While the number of participants is not big enough to make general assumptions, the overwhelming enthusiasm that a crowd composed of experts has shown is a good indicator of potential confirmation. Future studies will have to tackle potential biases such as these mentioned above and compare users’ satisfaction towards haptic devices that are highly specialized against those that can be adapted to multiple purposes. Also, there is a surprising lack of studies on users’ preferences on interactions metaphors: while natural interaction seems to be a logical choice, there are contexts in which it may be preferred to create new metaphors that make given tasks easier for the user. Answers need to be found to produce the best possible blending between haptics and VR.