Identify autism spectrum disorder via dynamic filter and deep spatiotemporal feature extraction

Highlights

• An effective model is proposed to identify children with autism spectrum disorder.

• Dynamic filters are introduced to customize feature maps for each scanpath.

• Field of view (FoV) maps are proposed to extract fixation-specific features.

• A module based on scanpath and saliency prediction is proposed to generate spatiotemporal features.

Abstract

Early intervention and treatment are crucial for individuals with autism spectrum disorder (ASD). However, it is challenging to identify individuals with ASD at an early age, i.e. under 3 years old, due to the lack of an effective and objective identification method. The mainstream clinical diagnosis relies on long-term observation of children’s behaviors, which is time-consuming and expensive, and thus how to accurately and quickly distinguish children with ASD in early childhood has become a critical issue. In this paper, we propose an eye movement based model to identify children with ASD. Specifically, children are required to freely observe some images. At the same time, their eye movements are recorded to analyze. Both the observed image and eye movements are input into our model. The input data are processed by the embedding layer, dynamic filters and LSTM block, respectively. Eventually, the spatiotemporal features are extracted to identify the eye movements belonging to a child with ASD or a typically developed child. Experiments on the Saliency4ASD dataset demonstrate that the proposed model achieves state-of-the-art performance in identifying children with ASD.

Introduction

ASD is a heritable neurodevelopmental disorder across the lifespan [1]. Since people with ASD show delays in development and cannot take care of themselves, the disorder has a remarkably negative impact on the family. The currently accepted and effective treatment is to intervene at an early age [2], so how to accurately and quickly diagnose children with ASD in early childhood has become a critical issue. Its diagnosis, however, is quite difficult because of its complicated causes. At present, the diagnosis mainly depends on the subjective and time-consuming clinical judgment, e.g. observation [3] and interview [4], which is limited by scarce medical services and tends to be inconsistent. Thus there is an urgent requirement for an effective and objective method to identify individuals with ASD.

Researchers applied eye trackers to collect eye movement data of the subjects when viewing a set of images, and found that the gazed points (namely fixations) significantly differed between the neurodevelopmental disorder group and the typically developed group, which indicates that individuals with neurodevelopmental disorders have an atypical visual attention pattern [5], [6]. Individuals with ASD prefer idiosyncratic objects (e.g. metalwork and keyboards) and hand-related utensils (e.g. scissors and bottles) [7], [8], [9]. They, on the contrary, tend to avoid eye contact with humans and have reduced attention to faces [10], [11]. Consequently, eye movement data have been used as a biomarker to identify individuals with ASD [12], [13].

In recent years, the rapid development of deep learning has remarkably shaped many research directions, and also contributed to the improvement in some eye movement related researches. Since deep learning has an inevitable demand for data, some eye movement datasets have been built and released [14], [15], [16]. Besides, there are also eye movement datasets that focus on some specific populations, such as individuals with ASD [17], [18] and people of different ages [19]. The success of deep learning is also pushing ASD identification forward. For example, by using the deep learning based face detection models, the eye movement data on face areas is analyzed and exploited to identify individuals with ASD [20]. Similarly, the automatically predicted saliency values at fixation locations are exploited to predict the fixation sequence belonging to either an ASD or a typical development (TD) [21], [22]. Besides, the deep visual features, which are extracted by well-trained deep neural networks, can be combined with eye movement data to identify people with ASD [23], [24]. However, due to the inadequate mining of eye movement data characteristics and the insufficiency of combining eye movement data with deep features, the accuracy of identification methods remains to be improved.

The proposed model takes a scanpath that consists of a sequence of fixations and the viewed image as inputs, and then predicts the viewer as either an ASD or a TD. Specifically, the scanpath is used repeatedly to fully capture the visual behavior pattern of an observer and is combined with the visual feature maps extracted by a deep neural network (DNN) to better identify people with ASD. In summary, the contributions of this paper can be summarized in two-fold:

• We propose to apply the dynamic filters to convolve with the deep visual feature maps, so as to transfer the universal feature maps, which are extracted from the observed image by pre-trained visual feature encoder, to sample-specific feature maps. The dynamic filter generator learns to process eye movement data to dynamic convolution kernels which can adjust responses in feature maps according to the eye movement.

• We propose a module to effectively extract spatiotemporal features of scanpath via the adoption of field of view (FoV) maps. The spatial features are extracted based on the field of view of each fixation and deep feature maps, and the temporal features are processed based on the spatial features by using an LSTM (Long Short-Term Memory) block. The spatiotemporal features, which vary as the observer’s attention shifts, are discriminative for ASD identification.

The rest of this paper is organized as follows. The related works on ASD identification and the application of the dynamic filter are introduced in Section 2. The proposed model is elaborated in Section 3. Experiment setup and results analysis are presented in Section 4. Section 5 summarizes the conclusions.