Haptics, audio and human–computer interaction are three scientific disciplines that share interests, issues and methodologies. Despite these common points, interaction between these communities are sparse, because each of them have their own publication venues, meeting places, etc. A venue to foster interaction between these three communities was created in 2006, the Haptic and Audio Interaction Design

In this contribution we studied how different rendering styles of a virtual human impacted users’ visual attention in an interactive medical training simulator. In a mixed design experiment, 78 participants interacted with a virtual human representing a sample from the non-photorealistic (NPR) to the photorealistic (PR) rendering continuity. We presented five rendering style samples scenarios, namely

Major usability and technical challenges have created mistrust of the potential of brain computer interfaces used to control video games in challenging environments like healthcare. Despite several studies showing low cost commercial headsets can read the brainwave patterns of its users with great potential for long term adoption; there are limited studies showing its efficacy in concrete healthcare

The integration of vibrotactile feedback in digital music instruments (DMIs) is thought to improve the instrument’s response and make it more suitable for expert musical interactions. However, given the extreme requirements of musical performances, there is a need for solutions allowing for independent control of frequency and amplitude over a wide frequency bandwidth (40–1000 Hz) and low harmonic

Remote collaboration using mixed reality (MR) enables two separated workers to collaborate by sharing visual cues. A local worker can share his/her environment to the remote worker for a better contextual understanding. However, prior techniques were using either 360 video sharing or a complicated 3D reconstruction configuration. This limits the interactivity and practicality of the system. In this

Persons who have survived a stroke might lower the risk of having recurrent strokes by adopting a healthier lifestyle with more exercise. One way to promote exercising is by fitness or exergame apps for mobile phones. Health and fitness apps are used by a significant portion of the consumers, but these apps are not targeted to stroke survivors, who may experience cognitive limitations (like fatigue

In this paper, the psychophysical abilities and limitations of the auditory and vibrotactile modality will be discussed. A direct comparison reveals similarities and differences. The knowledge of those is the basis for the design of perceptually optimized auditory-tactile human–machine interfaces or multimodal music applications. Literature data and own results for psychophysical characteristics are

A psychiatric diagnosis involves the physician’s ability to create an empathic interaction with the patient in order to accurately extract symptomatology (i.e., clinical manifestations). Virtual patients (VPs) can be used to train these skills but need to propose a structured and multimodal interaction situation, in order to simulate a realistic psychiatric interview. In this study we present a simulated

Many researchers have studied various visual communication cues (e.g. pointer, sketch, and hand gesture) in Mixed Reality remote collaboration systems for real-world tasks. However, the effect of combining them has not been so well explored. We studied the effect of these cues in four combinations: hand only, hand + pointer, hand + sketch, and hand + pointer + sketch, within the two user studies when

Crossmodal correspondence, a perceptual phenomenon which has been extensively studied in cognitive science, has been shown to play a critical role in people’s information processing performance. However, the evidence has been collected mostly based on strictly-controlled stimuli and displayed in a noise-free environment. In real-world interaction scenarios, background noise may blur crossmodal effects

An experiment is described that tested the possibility to classify wooden, plastic, and metallic objects based on reproduced auditory and vibrotactile stimuli. The results show that recognition rates are considerably above chance level with either unimodal auditory or vibrotactile feedback. Supported by those findings, the possibility to render virtual buttons for professional appliances with different

While the research area of artificial intelligence benefited from increasingly sophisticated machine learning techniques in recent years, the resulting systems suffer from a loss of transparency and comprehensibility, especially for end-users. In this paper, we explore the effects of incorporating virtual agents into explainable artificial intelligence (XAI) designs on the perceived trust of end-users

Interacting with an embodied conversational agent (ECA) in a professional context addresses social considerations to satisfy customer-relationship. This paper presents an experimental study about the perception of virtual intimacy in human-ECA interactions. We explore how an ECA’s multimodal communication affects our perception of virtual intimacy. To this end, we developed a virtual Tourism Information

Augmented reality (AR) technologies provide a shared platform for users to collaborate in a physical context involving both real and virtual content. To enhance the quality of interaction between AR users, researchers have proposed augmenting users’ interpersonal space with embodied cues such as their gaze direction. While beneficial in achieving improved interpersonal spatial communication, such shared

The nature of human/instrument interaction is a long-standing area of study, drawing interest from fields as diverse as philosophy, cognitive sciences, anthropology, human–computer-interaction, and artistic creation. In particular, the case of the interaction between performer and musical instrument provides an enticing framework for studying the instrumental dynamics that allow for embodiment, skill

Collaborative Mixed Reality (MR) technologies enable remote people to work together by sharing communication cues intrinsic to face-to-face conversations, such as eye gaze and hand gestures. While the role of visual cues has been investigated in many collaborative MR systems, the use of spatial auditory cues remains underexplored. In this paper, we present an MR remote collaboration system that shares

Human-friendly interactive features are preferred for service robots used in emerging areas of robotic applications such as caretaking, health care, assistance, education and entertainment since they are intended to be operated by non-expert users. Humans prefer to use voice instructions, responses, and suggestions in their daily interactions. Such voice instructions and responses often include uncertain

An extensive metamorphosis is currently taking place in the education industry due to the rapid adoption of different technologies and the proliferation of new student-instructor and student–student interaction models. While traditional face-to-face interaction is still the norm, mobile, online and virtual augmentations are increasingly adopted worldwide. Moreover, with the advent of gaming technology

Automatic detection of an individual’s mind-wandering state has implications for designing and evaluating engaging and effective learning interfaces. While it is difficult to differentiate whether an individual is mind-wandering or focusing on the task only based on externally observable behavior, brain-based sensing offers unique insights to internal states. To explore the feasibility, we conducted

Data analytics-driven solutions are widely used in various intelligent systems, where humans and machines make decisions collaboratively based on predictions. Human factors such as personality and trust have significant effects on such human–machine collaborations. This paper investigates effects of personality traits on user trust in human–machine collaborations under uncertainty and cognitive load

For almost 3 decades, research on auditory displays and sonification has been well advanced. Now, the auditory display community has arrived at the stage of sonic information design with a more systematic, refined necessity, going beyond random mappings between the referents and sounds. Due to its innate transdisciplinary nature of auditory display, it would be difficult to unify the methods to study

This paper describes a new web-based system specially adapted to the education of Czech pupils with visual impairment. The system integrates speech and language technologies with a web framework in lower secondary education, especially in mathematics and physics subjects. A new interface utilized the text-to-speech (TTS) synthesis for online automatic reading of educational texts. The interface provides

Abstract When designing movement sonifications, their effect on people’s movement must be considered. Recent work has shown how real-time sonification can be designed to alter the way people move. However, the mechanisms through which these sonifications alter people’s expectations of their movement is not well explained. This is especially important when considering musical sonifications, to which

Abstract Sonification has the potential to communicate a variety of data types to listeners including not just cognitive information, but also emotions and aesthetics. The goal of our dancer sonification project is to “sonify emotions as well as motions” of a dance performance via musical sonification. To this end, we developed and evaluated sonification strategies for adding a layer of emotional mappings

Abstract Emotion recognition based on facial expression is a challenging research topic and has attracted a great deal of attention in the past few years. This paper presents a novel method, utilizing multi-modal strategy to extract emotion features from facial expression images. The basic idea is to combine the low-level empirical feature and the high-level self-learning feature into a multi-modal

Abstract Numerous projects, normally run by younger people, are exploring robot use by older people. But are older any different from younger people in the way they want to interact with robots? Understanding older compared to younger people’s preferences will give researchers more insight into good design. We compared views on multi-modal human–robot interfaces, of older people living independently

Abstract Listening skills are critical for human communication. Social skills training (SST), performed by human trainers, is a well-established method for obtaining appropriate skills in social interaction. Previous work automated the process of social skills training by developing a dialogue system that teaches speaking skills through interaction with a computer agent. Even though previous work that

Abstract Flying an aircraft is a mentally demanding task where pilots must process a vast amount of visual, auditory and vestibular information. They have to control the aircraft by pulling, pushing and turning different knobs and levers, while knowing that mistakes in doing so can have fatal outcomes. Therefore, attempts to improve and optimize these interactions should not increase pilots’ mental

Abstract In this paper, we explore how the audio respiration signal can contribute to multimodal analysis of movement qualities. Within this aim, we propose two novel techniques which use the audio respiration signal captured by a standard microphone placed near to mouth and supervised machine learning algorithms. The first approach consists of the classification of a set of acoustic features extracted

Abstract Modern infotainment systems in automobiles facilitate driving at the cost of secondary tasks in addition to the primary task of driving. These secondary tasks have considerable chance to distract a driver from his primary driving task, thereby reducing safety or increasing cognitive workload. This paper presents an intelligent interactive head up display (HUD) on the windscreen of the driver

Abstract In the Arab world, mobile health (mHealth) applications are an effective way to provide health benefits to medically needy in the absence of health services. However, end users around the world use technology to perform tasks in a way that appears more natural, and closer to their cultural and personal preferences. Evidence from prior studies shows that culture is a vital factor in the success

Abstract With the popularity of immersive media, developing usable tools for content development is important for the production process. In the context of 3D audio production, user interfaces for authoring and editing 3D audio trajectories enable content developers, composers, practitioners, and recording and mixing engineers to define how audio sources travel in time. However, common interaction

Abstract Wi-Fi and acoustic signal-based human–computer interaction (HCI) methods have received growing attention in academia. However, there still are issues to be addressed despite their flourishing. In this work, we evaluate the practicality of the state-of-the-art signal-based HCI research in terms of the following six aspects—granularity, robustness, usability, efficiency, stability, and deployability

Abstract The term ‘multimodal’ typically refers to the combination of two or more sensory modalities; however, through the advancement of technology, modality variations within specific sensory systems are being discovered and compared in regards to physiological perception and response. The ongoing evaluation of air vs bone conduction auditory perception modalities is one such comparison. Despite

Abstract This paper presents the design and evaluation of four sonification methods to support monitoring and diagnosis in Electrocardiography (ECG). In particular we focus on an ECG abnormality called ST-elevation which is an important indicator of a myocardial infarction. Since myocardial infarction represents a life-threatening condition it is of essential value to detect an ST-elevation as early

Abstract This is a theoretical paper that considers the mapping problem, a foundational issue which arises when designing a sonification, as it applies to sonic information design. We argue that this problem can be addressed by using models from the field of embodied cognitive science, including embodied image schema theory, conceptual metaphor theory and conceptual blends, and from research which

Abstract We present a sonification method which we call Focused Audification (FA; previously: Augmented Audification) that allows to expand pure audification in a flexible way. It is based on a combination of single-side-band modulation and a pitch modulation of the original data stream. Based on two free parameters, the sonification’s frequency range is adjustable to the human hearing range and allows

The original version of this article unfortunately contained mistakes. The presentation order of Fig 5 and Fig. 6 was incorrect. The plots should have been presented according to the order of the sections in the text; the “Mean Task Duration” plot should have been presented first, followed by the “Perceived Intuitiveness” plot.

Abstract Jitter is an inevitable by-product of gaze detection. Because of this, gaze typing tends to be a slow and frustrating process. In this paper, we propose SliceType, a soft keyboard that is optimized for gaze input. Our main design objective is to use the screen area more efficiently by allocating a larger area to the target keys. We achieve this by determining the keys that will not be used

Abstract In this paper we present a study on the effects of auditory- and haptic feedback in a virtual throwing task performed with a point-based haptic device. The main research objective was to investigate if and how task performance and perceived intuitiveness is affected when interactive sonification and/or haptic feedback is used to provide real-time feedback about a movement performed in a 3D

Abstract The availability of additional force cues in haptic devices are often expected to improve control performance, over conditions that only provide visual feedback. However, there is little empirical evidence to show this to be true for the teleoperation control of remote vehicles [i.e., multiple unmanned aerial vehicles (UAVs)]. In this paper, the contribution of haptic force feedback cues to

Abstract In complex visual representations, there are several possible challenges for the visual perception that might be eased by adding sound as a second modality (i.e. sonification). It was hypothesized that sonification would support visual perception when facing challenges such as simultaneous brightness contrast or the Mach band phenomena. This hypothesis was investigated with an interactive