This article reports the effectiveness of high frame rate facial animated avatar and voice transformer in eLearning. Three avatars: (real male professor, male avatar, female avatar) were combined with male professor’s voice or VT-4 vocoder transformed voice to create six distinguished videos which were then viewed by university freshmen students. A total of 186 students divided into 15 groups participated

Remote collaboration has been studied for more than two decades and now there is the possibilities for new types of collaboration with the recent advances in immersive technologies such as Virtual, Augmented, Mixed Reality (VR/AR/MR). However, despite the increasing research interest in remote collaboration study with VR/AR/MR technologies, there is still a lack of academic venues specifically focusing

Exergames are serious games that involve physical exertion and are thought of as a form of exercise by using novel input models. Exergames are promising in improving the vestibular differences of children with autism but often lack of adaptation mechanisms that adjust the difficulty level of the exergame. In this paper, we present the design and development of Circus in Motion, a multimodal exergame

Haptics, audio and human–computer interaction are three scientific disciplines that share interests, issues and methodologies. Despite these common points, interaction between these communities are sparse, because each of them have their own publication venues, meeting places, etc. A venue to foster interaction between these three communities was created in 2006, the Haptic and Audio Interaction Design

In this contribution we studied how different rendering styles of a virtual human impacted users’ visual attention in an interactive medical training simulator. In a mixed design experiment, 78 participants interacted with a virtual human representing a sample from the non-photorealistic (NPR) to the photorealistic (PR) rendering continuity. We presented five rendering style samples scenarios, namely

Major usability and technical challenges have created mistrust of the potential of brain computer interfaces used to control video games in challenging environments like healthcare. Despite several studies showing low cost commercial headsets can read the brainwave patterns of its users with great potential for long term adoption; there are limited studies showing its efficacy in concrete healthcare

The integration of vibrotactile feedback in digital music instruments (DMIs) is thought to improve the instrument’s response and make it more suitable for expert musical interactions. However, given the extreme requirements of musical performances, there is a need for solutions allowing for independent control of frequency and amplitude over a wide frequency bandwidth (40–1000 Hz) and low harmonic

Remote collaboration using mixed reality (MR) enables two separated workers to collaborate by sharing visual cues. A local worker can share his/her environment to the remote worker for a better contextual understanding. However, prior techniques were using either 360 video sharing or a complicated 3D reconstruction configuration. This limits the interactivity and practicality of the system. In this

Persons who have survived a stroke might lower the risk of having recurrent strokes by adopting a healthier lifestyle with more exercise. One way to promote exercising is by fitness or exergame apps for mobile phones. Health and fitness apps are used by a significant portion of the consumers, but these apps are not targeted to stroke survivors, who may experience cognitive limitations (like fatigue

A psychiatric diagnosis involves the physician’s ability to create an empathic interaction with the patient in order to accurately extract symptomatology (i.e., clinical manifestations). Virtual patients (VPs) can be used to train these skills but need to propose a structured and multimodal interaction situation, in order to simulate a realistic psychiatric interview. In this study we present a simulated

In this paper, the psychophysical abilities and limitations of the auditory and vibrotactile modality will be discussed. A direct comparison reveals similarities and differences. The knowledge of those is the basis for the design of perceptually optimized auditory-tactile human–machine interfaces or multimodal music applications. Literature data and own results for psychophysical characteristics are

Many researchers have studied various visual communication cues (e.g. pointer, sketch, and hand gesture) in Mixed Reality remote collaboration systems for real-world tasks. However, the effect of combining them has not been so well explored. We studied the effect of these cues in four combinations: hand only, hand + pointer, hand + sketch, and hand + pointer + sketch, within the two user studies when

Crossmodal correspondence, a perceptual phenomenon which has been extensively studied in cognitive science, has been shown to play a critical role in people’s information processing performance. However, the evidence has been collected mostly based on strictly-controlled stimuli and displayed in a noise-free environment. In real-world interaction scenarios, background noise may blur crossmodal effects

An experiment is described that tested the possibility to classify wooden, plastic, and metallic objects based on reproduced auditory and vibrotactile stimuli. The results show that recognition rates are considerably above chance level with either unimodal auditory or vibrotactile feedback. Supported by those findings, the possibility to render virtual buttons for professional appliances with different

While the research area of artificial intelligence benefited from increasingly sophisticated machine learning techniques in recent years, the resulting systems suffer from a loss of transparency and comprehensibility, especially for end-users. In this paper, we explore the effects of incorporating virtual agents into explainable artificial intelligence (XAI) designs on the perceived trust of end-users

Interacting with an embodied conversational agent (ECA) in a professional context addresses social considerations to satisfy customer-relationship. This paper presents an experimental study about the perception of virtual intimacy in human-ECA interactions. We explore how an ECA’s multimodal communication affects our perception of virtual intimacy. To this end, we developed a virtual Tourism Information

Augmented reality (AR) technologies provide a shared platform for users to collaborate in a physical context involving both real and virtual content. To enhance the quality of interaction between AR users, researchers have proposed augmenting users’ interpersonal space with embodied cues such as their gaze direction. While beneficial in achieving improved interpersonal spatial communication, such shared

The nature of human/instrument interaction is a long-standing area of study, drawing interest from fields as diverse as philosophy, cognitive sciences, anthropology, human–computer-interaction, and artistic creation. In particular, the case of the interaction between performer and musical instrument provides an enticing framework for studying the instrumental dynamics that allow for embodiment, skill

Collaborative Mixed Reality (MR) technologies enable remote people to work together by sharing communication cues intrinsic to face-to-face conversations, such as eye gaze and hand gestures. While the role of visual cues has been investigated in many collaborative MR systems, the use of spatial auditory cues remains underexplored. In this paper, we present an MR remote collaboration system that shares

Human-friendly interactive features are preferred for service robots used in emerging areas of robotic applications such as caretaking, health care, assistance, education and entertainment since they are intended to be operated by non-expert users. Humans prefer to use voice instructions, responses, and suggestions in their daily interactions. Such voice instructions and responses often include uncertain

An extensive metamorphosis is currently taking place in the education industry due to the rapid adoption of different technologies and the proliferation of new student-instructor and student–student interaction models. While traditional face-to-face interaction is still the norm, mobile, online and virtual augmentations are increasingly adopted worldwide. Moreover, with the advent of gaming technology

Automatic detection of an individual’s mind-wandering state has implications for designing and evaluating engaging and effective learning interfaces. While it is difficult to differentiate whether an individual is mind-wandering or focusing on the task only based on externally observable behavior, brain-based sensing offers unique insights to internal states. To explore the feasibility, we conducted

Data analytics-driven solutions are widely used in various intelligent systems, where humans and machines make decisions collaboratively based on predictions. Human factors such as personality and trust have significant effects on such human–machine collaborations. This paper investigates effects of personality traits on user trust in human–machine collaborations under uncertainty and cognitive load

For almost 3 decades, research on auditory displays and sonification has been well advanced. Now, the auditory display community has arrived at the stage of sonic information design with a more systematic, refined necessity, going beyond random mappings between the referents and sounds. Due to its innate transdisciplinary nature of auditory display, it would be difficult to unify the methods to study

This paper describes a new web-based system specially adapted to the education of Czech pupils with visual impairment. The system integrates speech and language technologies with a web framework in lower secondary education, especially in mathematics and physics subjects. A new interface utilized the text-to-speech (TTS) synthesis for online automatic reading of educational texts. The interface provides

The original version of this article unfortunately contained mistakes. The presentation order of Fig 5 and Fig. 6 was incorrect. The plots should have been presented according to the order of the sections in the text; the “Mean Task Duration” plot should have been presented first, followed by the “Perceived Intuitiveness” plot.

When designing movement sonifications, their effect on people’s movement must be considered. Recent work has shown how real-time sonification can be designed to alter the way people move. However, the mechanisms through which these sonifications alter people’s expectations of their movement is not well explained. This is especially important when considering musical sonifications, to which people bring

Sonification has the potential to communicate a variety of data types to listeners including not just cognitive information, but also emotions and aesthetics. The goal of our dancer sonification project is to “sonify emotions as well as motions” of a dance performance via musical sonification. To this end, we developed and evaluated sonification strategies for adding a layer of emotional mappings to

The term ‘multimodal’ typically refers to the combination of two or more sensory modalities; however, through the advancement of technology, modality variations within specific sensory systems are being discovered and compared in regards to physiological perception and response. The ongoing evaluation of air vs bone conduction auditory perception modalities is one such comparison. Despite an increased

This paper presents the design and evaluation of four sonification methods to support monitoring and diagnosis in Electrocardiography (ECG). In particular we focus on an ECG abnormality called ST-elevation which is an important indicator of a myocardial infarction. Since myocardial infarction represents a life-threatening condition it is of essential value to detect an ST-elevation as early as possible

This is a theoretical paper that considers the mapping problem, a foundational issue which arises when designing a sonification, as it applies to sonic information design. We argue that this problem can be addressed by using models from the field of embodied cognitive science, including embodied image schema theory, conceptual metaphor theory and conceptual blends, and from research which treats sound

We present a sonification method which we call Focused Audification (FA; previously: Augmented Audification) that allows to expand pure audification in a flexible way. It is based on a combination of single-side-band modulation and a pitch modulation of the original data stream. Based on two free parameters, the sonification’s frequency range is adjustable to the human hearing range and allows to interactively

Modern infotainment systems in automobiles facilitate driving at the cost of secondary tasks in addition to the primary task of driving. These secondary tasks have considerable chance to distract a driver from his primary driving task, thereby reducing safety or increasing cognitive workload. This paper presents an intelligent interactive head up display (HUD) on the windscreen of the driver that does

With the popularity of immersive media, developing usable tools for content development is important for the production process. In the context of 3D audio production, user interfaces for authoring and editing 3D audio trajectories enable content developers, composers, practitioners, and recording and mixing engineers to define how audio sources travel in time. However, common interaction techniques

Wi-Fi and acoustic signal-based human–computer interaction (HCI) methods have received growing attention in academia. However, there still are issues to be addressed despite their flourishing. In this work, we evaluate the practicality of the state-of-the-art signal-based HCI research in terms of the following six aspects—granularity, robustness, usability, efficiency, stability, and deployability

Listening skills are critical for human communication. Social skills training (SST), performed by human trainers, is a well-established method for obtaining appropriate skills in social interaction. Previous work automated the process of social skills training by developing a dialogue system that teaches speaking skills through interaction with a computer agent. Even though previous work that simulated

Numerous projects, normally run by younger people, are exploring robot use by older people. But are older any different from younger people in the way they want to interact with robots? Understanding older compared to younger people’s preferences will give researchers more insight into good design. We compared views on multi-modal human–robot interfaces, of older people living independently, with students

Mobile users lose a lot of time on their smartphones. They interact even with busy hands (hands-free interactions), distracted eyes (eyes-free interactions) and in different life situations (while walking, eating, working, etc.). The Time Well Spent (TWS) is a movement that aims to design applications which respect the users choices and availability. In this paper, we discuss how the multimodal mobile

In the Arab world, mobile health (mHealth) applications are an effective way to provide health benefits to medically needy in the absence of health services. However, end users around the world use technology to perform tasks in a way that appears more natural, and closer to their cultural and personal preferences. Evidence from prior studies shows that culture is a vital factor in the success of a

Flying an aircraft is a mentally demanding task where pilots must process a vast amount of visual, auditory and vestibular information. They have to control the aircraft by pulling, pushing and turning different knobs and levers, while knowing that mistakes in doing so can have fatal outcomes. Therefore, attempts to improve and optimize these interactions should not increase pilots’ mental workload

During the last two decades, development of 3D object selection techniques has been widely studied because it is critical for providing an interactive virtual environment to users. Previous techniques encounter difficulties with selecting small or distant objects, as well as naturalness and physical fatigue. Although eye-hand based interaction techniques have been promoted as the ideal solution to

In complex visual representations, there are several possible challenges for the visual perception that might be eased by adding sound as a second modality (i.e. sonification). It was hypothesized that sonification would support visual perception when facing challenges such as simultaneous brightness contrast or the Mach band phenomena. This hypothesis was investigated with an interactive sonification

Emotion recognition based on facial expression is a challenging research topic and has attracted a great deal of attention in the past few years. This paper presents a novel method, utilizing multi-modal strategy to extract emotion features from facial expression images. The basic idea is to combine the low-level empirical feature and the high-level self-learning feature into a multi-modal feature

Hand gesture is a kind of natural interaction way and hand gesture recognition has recently become more and more popular in human–computer interaction. However, the complexity and variations of hand gesture like various illuminations, views, and self-structural characteristics make the hand gesture recognition still challengeable. How to design an appropriate feature representation and classifier are

This paper presents an object acquisition and selection approach in human computer interaction systems. In this approach, objects placed over computer screen are automatically scanned and the user performs voluntary eye blinks for object selection when the focus comes over the object of interest. Here, scanning means moving the focus over objects placed on the computer screen one by one and the scanning

In this paper, we explore how the audio respiration signal can contribute to multimodal analysis of movement qualities. Within this aim, we propose two novel techniques which use the audio respiration signal captured by a standard microphone placed near to mouth and supervised machine learning algorithms. The first approach consists of the classification of a set of acoustic features extracted from

In this paper we investigate how natural language interfaces can be integrated with cars in a way such that their influence on driving performance is being minimized. In particular, we focus on how speech-based interaction can be supported through a visualization of the conversation. Our work is motivated by the fact that speech interfaces (like Alexa, Siri, Cortana, etc.) are increasingly finding

In the last decade, the number and variety of secondary tasks in modern vehicles has grown exponentially. To address this variety, drivers can choose between alternative input modalities to complete each task in the most adequate way. However, the process of switching between different modalities might cause increased cognitive effort and finally result in a loss of efficiency. Therefore, the effects

Currently, lane change decision aid systems primarily address foveal vision and thus compete for drivers’ attention with interfaces of other assistant systems. Also, alternative modalities such as acoustic perception (Mahapatra et al., in: 2008 International conference on advanced computer theory and engineering, pp 992–995. https://doi.org/10.1109/ICACTE.2008.165, 2008), tactile perception (Löcken

The general role of personal assistants in form of anthropomorphised conversational, virtual or robotic agents in cars is subject to research since a few years and the first results indicate numerous positive effects of these anthropomorphised interfaces. However, no comprehensive review of the conducted studies has been comprised yet. Furthermore, existing studies on the effect of anthropomorphism

Using touchscreens while driving introduces competition for visual attention that increases crash risk. To resolve this issue, we have developed an auditory-supported air gesture system. We conducted two experiments using the driving simulator to investigate the influence of this system on driving performance, eye glance behavior, secondary task performance, and driver workload. In Experiment 1 we

Emotions influence the way drivers process and react to internal or environmental factors, but relatively little research has focused on drivers’ emotions. Of many emotional states, anger is considered the most serious threat on the road. Therefore, having an affective intelligent system in the car that can estimate drivers’ anger and respond to it appropriately can help drivers adapt to moment-to-moment

In this article, we describe methods and consequences for giving audience members interactive control over the real-time sonification of performer movement data in electronic music performance. We first briefly describe how to technically implement a musical performance in which each audience member can interactively construct and change their own individual sonification of performers’ movements, heard

The usability of virtual keyboard based eye-typing systems is currently limited due to the lack of adaptive and user-centered approaches leading to low text entry rate and the need for frequent recalibration. In this work, we propose a set of methods for the dwell time adaptation in asynchronous mode and trial period in synchronous mode for gaze based virtual keyboards. The rules take into account

The availability of additional force cues in haptic devices are often expected to improve control performance, over conditions that only provide visual feedback. However, there is little empirical evidence to show this to be true for the teleoperation control of remote vehicles [i.e., multiple unmanned aerial vehicles (UAVs)]. In this paper, the contribution of haptic force feedback cues to the teleoperation

The way doctors deliver bad news has a significant impact on the therapeutic process. In order to facilitate doctor’s training, we have developed an embodied conversational agent simulating a patient to train doctors to break bad news. In this article, we present an evaluation of the virtual reality training platform comparing the users’ experience depending on the virtual environment displays: a PC

It is crucial that teaching methods are conducted accurately when teaching children with autism spectrum disorder. This can be addressed by providing new teachers with ample opportunity to practice their teaching skills while receiving feedback from a supervisor. Unfortunately, due to lack of resources, this is not always possible. Advances in Virtual Reality and Virtual Human research are making technical

Social interactions entail reciprocal reactions where one’s communicative acts triggers responses in others. Fluent interpersonal exchange relies on the ability to discriminate behaviors produced by others that are responses to one’s actions, thus involving a social sense of agency. Given the pivotal role of gaze in human communication, we propose to use gaze following as a model for studying the sense

When engaging in social interaction, people rely on their ability to reason about unobservable mental content of others, which includes goals, intentions, and beliefs. This so-called theory of mind ability allows them to more easily understand, predict, and influence the behavior of others. People even use their theory of mind to reason about the theory of mind of others, which allows them to understand

Jitter is an inevitable by-product of gaze detection. Because of this, gaze typing tends to be a slow and frustrating process. In this paper, we propose SliceType, a soft keyboard that is optimized for gaze input. Our main design objective is to use the screen area more efficiently by allocating a larger area to the target keys. We achieve this by determining the keys that will not be used for the