Solving visual question answering (VQA) task requires recognizing many diverse visual concepts as the answer. These visual concepts contain rich structural semantic meanings, e.g., some concepts in VQA are highly related (e.g., red & blue), some of them are less relevant (e.g., red & standing). It is very natural for humans to efficiently learn concepts by utilizing their semantic meanings to concentrate

Understanding multiparty human interaction dynamics is a challenging problem involving multiple data modalities and complex ordered interactions between multiple people. We propose a unified framework that integrates synchronized video, audio, and text streams from four people to capture the interaction dynamics in natural group meetings. We focus on estimating the dynamic social role of the meeting

Speech recognition and machine translation have made major progress over the past decades, providing practical systems to map one language sequence to another. Although multiple modalities such as sound and video are becoming increasingly available, the state-of-the-art systems are inherently unimodal, in the sense that they take a single modality — either speech or text — as input. Evidence from human

Presents the front cover for this issue of the publication.

Presents a listing of the editorial board, board of governors, current staff, committee members, and/or society editors for this issue of the publication.

Presents the table of contents for this issue of the publication.

Approximately one-fifth of theworld’s population suffer or have suffered from voice and speech production disorders due to diseases or some other dysfunction. Thus, there is a clear need for objective ways to evaluate the quality of voice and speech as well as its link to vocal fold activity, to evaluate the complex interaction between the larynx and voluntary movements of the articulators (i.e., lips

Obstructive Sleep Apnea (OSA) is a sleep breathing disorder affecting at least 3–7% of male adults and 2–5% of female adults between 30 and 70 years. It causes recurrent partial or total obstruction episodes at the level of the pharynx which causes cessation of breath during sleep. The number of obstruction episodes per sleep hour, known as Apnea-Hypopnea Index (AHI), along with the degree of the daytime

Obstructive sleep apnea (OSA) is a sleep disorder in which pharyngeal collapse during sleep causes complete (apnea) or partial (hypopnea) airway obstruction. OSA is common and can have severe implications, but often remains undiagnosed. The most widely used objective measure of OSA severity is the number of obstructive events per hour of sleep, known as the apnea-hypopnea index (AHI). This study reports

Clinical literature provides convincing evidence that language deficits in Alzheimer's disease (AD) allow for distinguishing patients with dementia from healthy subjects. Currently, computational approaches have widely investigated lexicosemantic aspects of discourse production, while pragmatic aspects like cohesion and coherence, are still mostly unexplored. In this article, we aim at providing a

Speech analysis could provide an indicator of Alzheimer's disease and help develop clinical tools for automatically detecting and monitoring disease progression. While previous studies have employed acoustic (speech) features for characterisation of Alzheimer's dementia, these studies focused on a few common prosodic features, often in combination with lexical and syntactic features which require transcription

It is widely accepted that information derived from analyzing speech (the acoustic signal) and language production (words and sentences) serves as a useful window into the health of an individual's cognitive ability. In fact, most neuropsychological testing batteries have a component related to speech and language where clinicians elicit speech from patients for subjective evaluation across a broad

The heterogeneity in Autism Spectrum Disorder (ASD) remains a challenging and unsolved issue in the current clinical practice. The behavioral differences between ASD subgroups are subtle and can be hard to be manually discerned by experts. Here, we propose a computational framework that is capable of modeling both vocal behaviors and body gestural movements of the interlocutors with their intricate

Information theoretic entropy measures are calculated from estimates of the probabilities of the constituent symbolic events. In natural sequences, such as those occurring in human language, the probabilistic structure typically follows a rank ordering pattern. Entropy has been used to model language by using large data sets to characterize the underlying source. To model the dynamic characteristics

Dysarthria is a motor speech impairment, often characterized by slow and slurred speech that is generally incomprehensible by human listeners. An understanding of the intelligibility level of the patient's dysarthric speech can provide an insight into the progression/status of the underlying cause and is essential for planning therapy. Automatic assessment of dysarthric speech intelligibility can be

Aphasia is a common type of acquired language impairment resulting from dysfunction in specific brain regions. Analysis of narrative spontaneous speech, e.g., story-telling, is an essential component of standardized clinical assessment on people with aphasia (PWA). Subjective assessment by trained speech-language pathologists (SLP) have many limitations in efficiency, effectiveness and practicality

Dysarthria is a speech-motor disorder that affects the articulatory systems inhibiting their speech communication efforts. To handle their communication problems, a speech recognition-based augmentative and alternative communication aid is used as an attractive alternative. However, successful development of an automatic speech recognition (ASR)-based aid depends on the availability of sufficient speech

The use of a psychoacoustic roughness model as a predictor of creaky voice is reported. We found that the roughness temporal profile of vocalic segments can predict the presence of creakiness in speech. Using a simple bi-directional Recurrent Neural Network ( rnn ), we were able to predict the presence of creakiness in vocalic segments from only roughness traces with an accuracy similar to that obtained

Automatic detection of voice pathology enables objective assessment and earlier intervention for the diagnosis. This study provides a systematic analysis of glottal source features and investigates their effectiveness in voice pathology detection. Glottal source features are extracted using glottal flows estimated with the quasi-closed phase (QCP) glottal inverse filtering method, using approximate

A new method for the automated speech production assessment (ASPA) of hearing impaired children is presented in this paper, providing feedback about the pronunciation quality of words and sentences uttered during unsupervised practice in the course of speech development. A database of the sounds produced by hearing impaired subjects was set up and assessed with a subjective test. The Mean Opinion Score

Recently, spectro-temporal representation of speech has been used in many fields of speech processing. Owing to this, we explore the use of spectro-temporal representation for speech intelligibility assessment especially for dysarthric speech. In this work, we investigate the use of spectro-temporal representations to evaluate intelligibility levels using artificial neural network (ANN) and convolutional

Given the limited accessibility to Speech and Language Pathologists (SLPs) children in need often have, pediatric Computer-Aided Speech Therapy (CAST) tools can play an important role in the early diagnosis and treatment of speech disorders. However, various challenges impede the implementation of accurate automated analysis of speech disorders in children. In this article, we first discuss three key

This article addresses the automatic assessment of voice quality according to the GRB scale, based on the use of a variety of deep learning architectures for prediction purposes. The proposed architectures are multimodal, because they employ multiples sources of information; and also multi-output, because they simultaneously predict all the traits of the GRB scale. A feature engineering approach is

Early interventions in mental health conditions such as Major Depressive Disorder (MDD) are critical to improved health outcomes, as they can help reduce the burden of the disease. As the efficient diagnosis of depression severity is therefore highly desirable, the use of behavioural cues such as speech characteristics in diagnosis is attracting increasing interest in the field of quantitative mental

The processing of speech as an explicit sequence of events is common in automatic speech recognition (linguistic events), but has received relatively little attention in paralinguistic speech classification despite its potential for characterizing broad acoustic event sequences. This paper proposes a framework for analyzing speech as a sequence of acoustic events, and investigates its application to

Subglottal air pressure plays a major role in voice production and is a primary factor in controlling voice onset, offset, sound pressure level, glottal airflow, vocal fold collision pressures, and variations in fundamental frequency. Previous work has shown promise for the estimation of subglottal pressure from an unobtrusive miniature accelerometer sensor attached to the anterior base of the neck

A technique is presented for detecting glottal excitation epochs in the speech signal, using Hilbert envelope for enhancing the glottal excitation and maximum-sum subarray for marking the epochs. The processing comprises the steps of dynamic range compression, Hilbert envelope calculation, saliency enhancing, and epoch marking. The dynamic range compression reduces the amplitude variation of the signal

Presents corrections to the above named paper.

Presents a listing of the editorial board, board of governors, current staff, committee members, and/or society editors for this issue of the publication.

Background noise, interfering speech and room reverberation frequently distort target speech in real listening environments. In this study, we address joint speech separation and dereverberation, which aims to separate target speech from background noise, interfering speech and room reverberation. In order to tackle this fundamentally difficult problem, we propose a novel multimodal network that exploits

Deep learning methods haverevolutionized speech recognition, image recognition, and natural language processing since 2010. Each of these tasks involves a single modality in their input signals. However, many applications in the artificial intelligence field involve multiple modalities. Therefore, it is of broad interest to study the more difficult and complex problem of modeling and learning across

Image classification models have achieved satisfactory performance on many datasets, sometimes even better than humans. However, the model attention is unclear since the lack of interpretability. This paper investigates the fidelity and interpretability of model attention. We propose an Explainable Attribute-based Multi-task (EAT) framework to concentrate the model attention on the discriminative image

Direct speech-to-image translation without text is an interesting and useful topic due to the potential applications in human-computer interaction, art creation, computer-aided design. etc. Not to mention that many languages have no writing form. However, as far as we know, it has not been well-studied how to translate the speech signals into images directly and how well they can be translated. In

This paper proposes a new strategy for learning effective cross-modal joint embeddings using self-supervision. We set up the problem as one of cross-modal retrieval, where the objective is to find the most relevant data in one domain given input in another. The method builds on the recent advances in learning representations from cross-modal self-supervision using contrastive or binary cross-entropy

The aggregate-label learning paradigm tackles the long-standing temporary credit assignment (TCA) problem in neuroscience and machine learning, enabling spiking neural networks to learn multimodal sensory clues with delayed feedback signals. However, the existing aggregate-label learning algorithms only work for single spiking neurons, and with low learning efficiency, which limit their real-world

Target speech separation refers to extracting a target speaker's voice from an overlapped audio of simultaneous talkers. Previously the use of visual modality for target speech separation has demonstrated great potentials. This work proposes a general multi-modal framework for target speech separation by utilizing all the available information of the target speaker, including his/her spatial location

Presents the front cover for this issue of the publication.

Presents a listing of the editorial board, board of governors, current staff, committee members, and/or society editors for this issue of the publication.

Presents the table of contents for this issue of the publication.

The papers in this special section examine perception driven 360 degrees video processing applications. Covers such topics as perception models, VQA metrics, and compression/streaming techniques for 360° video/image.

Nowadays, 360° video/image has been increasingly popular and drawn great attention. The spherical viewing range of 360° video/image accounts for huge data, which pose the challenges to 360° video/image processing in solving the bottleneck of storage, transmission, etc. Accordingly, the recent years have witnessed the explosive emergence of works on 360° video/image processing. In this article, we review

Panoramic videos are becoming more and more easily obtained for common users. Although these videos have $360^\circ$ field of view, they are usually displayed with perspective views, which needs the saliency informations for viewing angle selection. In this paper, we propose a saliency prediction network for $360^\circ$ videos. Our network takes video frames and optical flows in cube map format as

Image-based salient object detection (SOD) has been extensively explored in the past decades. However, SOD on 360 $^\circ$ omnidirectional images is less studied owing to the lack of datasets with pixel-level annotations. Toward this end, this paper proposes a 360 $^\circ$ image-based SOD dataset that contains 500 high-resolution equirectangular images. We collect the representative equirectangular

To render omnidirectional (or 360°) visual content, a projection that maps the pixels from a portion of the viewing sphere to a 2D plane must be employed; this projection creates the viewport image shown to the user and thus has an important role on the quality of experience, for this type of content. However, a sphere to planar projection always introduces geometrical distortions on the rendered image

360-degree images/videos have been dramatically increasing in recent years. The characteristic of omnidirectional-view results in high resolution of 360-degree images/videos, which makes them difficult to be transported and stored. To deal with the problem, video coding technologies are used to compress the omnidirectional content but they will introduce the compression distortion. Therefore, it is

Virtual reality (VR) videos, especially 360-degree VR videos, have attracted substantial research interests with the explosion of mobile VR devices in recent years. In this work, we propose a VR quality of experience (QoE) evaluation framework including online, offline and mixed scenarios that can meet the requirement of real applications. Under this framework, we further develop a subjective evaluation

Virtual Reality (VR) and its applications have attracted significant and increasing attention. However, the requirements of much larger file sizes, different storage formats, and immersive viewing conditions pose significant challenges to the goals of acquiring, transmitting, compressing and displaying high quality VR content. Towards meeting these challenges, it is important to be able to understand

Objective quality assessment of stereoscopic omnidirectional images is a challenging problem since it is influenced by multiple aspects such as projection deformation, field of view (FoV) range, binocular vision, visual comfort, etc. Existing studies show that classic 2D or 3D image quality assessment (IQA) metrics are not able to perform well for stereoscopic omnidirectional images. However, very

To provide excellent visual experience for customers, virtual reality (VR) sources require higher resolutions and better visual quality than traditional picture sequences. The content of a VR video can be mapped into a sphere by playing devices to present a $360^{\circ }$ scene, which is usually called VR360 in industrial community. The most popular formats for VR360 sources are the equirectangular

The 360-degree video is projected to 2-D formats using various projection methods for efficient compression. As a necessary part of general-video compression, rate control is also indispensable for the projected 360-degree video compression. However, the current rate control algorithm has not been optimized for the 360-degree video compression yet. The Coding Tree Unit (CTU) level bit allocation in

Viewport-Dependent Streaming (VDS) is a preferred way in practice to deliver the omnidirectional videos, of which a High-Quality (HQ) scale is applied for the content in current viewport but a Low-Quality (LQ) scale elsewhere. Quality adaptation or refinement happens after users stabilize their fixations to a new viewport. In this article, we formulate this as a perceptually optimized quality adaptation

In this article, we study the server-side rate adaptation problem for streaming tile-based adaptive 360-degree videos to multiple users who are competing for transmission resources at the network bottleneck. Specifically, we develop a convolutional neural network (CNN)-based viewpoint prediction model to capture the nonlinear relationship between the future and historical viewpoints. A Laplace distribution

The 360-degree video allows users to enjoy the whole scene by interactively switching viewports. However, the huge data volume of the 360-degree video limits its remote applications via network. To provide high quality of experience ( QoE ) for remote web users, this paper presents a tile-based adaptive streaming method for 360-degree videos. First, we propose a simple yet effective rate adaptation

The 360-degree video/image, also called an omnidirectional video/image or panoramic video/image, is very important in some emerging areas such as virtual reality (VR). Therefore, corresponding image generation algorithms are urgently needed. However, existing image generation models mainly focus on 2-D images and do not consider the spherical structures of panoramic images. In this article, we propose

360 $^{\circ }$ omnidirectional images are very helpful in creating immersive multimedia contents, which enables a huge demand in their efficient generation and effective assessment. In this paper, we leverage an attentive idea to meet this demand by addressing two concerns: how to generate a good omnidirectional image in a fast and robust way and what is a good omnidirectional image for human. To

Omnidirectional cameras have recently received significant attention in panoramic imaging systems such as virtual reality (VR) technology; however, the strong geometric distortion in omnidirectional images severely affects the object recognition and semantic understanding. In this paper, we propose an automatic omnidirectional image distortion correction approach powered by a unified learning model

Presents a listing of the editorial board, board of governors, current staff, committee members, and/or society editors for this issue of the publication.

In many signal processing and machine learning applications, datasets containing private information are held at different locations, requiring the development of distributed privacy-preserving algorithms. Tensor and matrix factorizations are key components of many processing pipelines. In the distributed setting, differentially private algorithms suffer because they introduce noise to guarantee privacy

Pinpointing the sources of dementia is crucial to the effective treatment of neurodegenerative diseases. In this paper, we propose a diffusion model with impulsive sources over the brain connectivity network to model the progression of brain atrophy. To reliably estimate the atrophy sources, we impose sparse regularization on the source distribution and solve the inverse problem with an efficient gradient

Functional connectivity describes neural activity from resting-state functional magnetic resonance imaging (rs-fMRI). This noninvasive modality is a promising imaging biomarker of neurodegenerative diseases, such as Alzheimer's disease (AD), where the connectome can be an indicator to assess and to understand the pathology. However, it only provides noisy measurements of brain activity. As a consequence