LSTM guided ensemble correlation filter tracking with appearance model pool

Highlights

• We propose adaptive aggregation of CNN features from multiple layers for tracking.

• The weights for aggregation are determined using LSTM.

• Filters are updated using an appearance model pool to prevent faulty updates.

• Experimental results reveal state of the art performance.

Abstract

Deep learning based visual trackers have the potential to provide good performance for object tracking. Most of them use hierarchical features learned from multiple layers of a deep network. However, issues related to deterministic aggregation of these features from various layers, difficulties in estimating variations in scale or rotation of the object being tracked, as well as challenges in effectively modeling the object’s appearance over long time periods leaves substantial scope to improve performance. In this paper, we propose a tracker that learns correlation filters over features from multiple layers of a VGG network. A correlation filter for an individual layer is used to predict the target location. We adaptively learn the contribution of an ensemble of correlation filters for the final location estimation using an LSTM. An adaptive approach is advantageous as different layers encode diverse feature representations and a uniform contribution would not fully exploit this contrastive information. To this end, we use an LSTM as it encodes the interactions for past appearances which is useful for tracking. Further, the scale and rotation parameters are estimated using respective correlation filters. Additionally, an appearance model pool is used that prevents the correlation filter from drifting. Experimental results achieved on five public datasets — Object Tracking Benchmark (OTB100), Visual Object Tracking (VOT) Benchmark 2016, VOT Benchmark 2017, Tracking Dataset and UAV123 Dataset, reveal that our approach outperforms state of the art approaches for object tracking.

Introduction

From video surveillance to human–computer interaction, visual object tracking is one of the most fundamental applications of computer vision. Recently, deep learning has been demonstrated to significantly improve tracking performance due to its rich feature representation. The majority of deep learning based trackers use pre-trained Convolutional Neural Networks (CNNs) that are trained for classification. They predict target location based on features extracted using the convolutional layers. However, the differing needs of classification and tracking algorithms mean that CNNs trained for classification often result in non-robust tracking. These difficulties are caused by the nature of the tracking problem, where video sequences offer multiple challenges like non-stationary cameras, simultaneous movement of the target and camera, environmental turbulence, non-uniform backgrounds, occlusions, background clutter and many more; often with these challenging conditions occurring at the same time. Hence, to construct a robust tracker with practical utility, one needs to simultaneously be able to overcome all of these challenges.

In this paper, we propose a deep learning based tracker that predicts the target location based on multi-layer convolutional feature aggregation, scale and rotation estimation as well as an appearance model pool. Compared to existing trackers that use the output of a single CNN layer for target location estimation, multi-layer trackers perform better (Qi et al., 2016, Ma et al., 2015a). This is because deep layers in a CNN capture only category-level semantic information, which is best suited to classification tasks. Tracking requires a feature representation that also contains spatial information for accurate target localization. Therefore, we combine activations from deep layers with those of earlier layers (that capture more spatial information), obtaining more reliable prediction. However, the prediction accuracy of each layer may vary from frame to frame and combining all the predictions with equal weight may result in inaccurate tracking. Thus, a reliable mechanism that can compute the contribution of each layer in estimating the target’s final location is required. To this end, we use a Long Short Term Memory (LSTM) network that computes the prediction accuracy of each layer, and which is used to assign a weight to each layer. The target’s final location is predicted as the weighted sum of predictions, using the above mentioned weights from different layers. However, during situations like illumination variation, occlusion or appearance change, ambiguous training samples may lead to a poor correlation filter update. This may result in tracking failure during longer sequences. To overcome this, it is imperative to introduce a corrective measure that can prevent faulty updates of the correlation filter. Thus, we introduce an appearance model pool that stores the best appearances of the object and provides the correlation filters with the best updating template during challenging scenarios. This prevents the tracker drifting during occlusion and other challenges, resulting in successful long term tracking. Further, during the course of tracking, the object can undergo scale and rotation variations. In order to address these variations, we use a strategy based on FHOG features (Felzenszwalb et al., 2010), that estimates target scale and rotation in each frame.

To summarize, our major technical contributions are as follows:

• We introduce an LSTM network that adaptively learns the weights to combine the target location predicted by each correlation filter. This helps in improving the aggregation of predictions from different layers.

• We introduce an appearance model based correction module that avoids faulty updates to the correlation filters. This promotes efficient long term tracking.

• To further improve the performance, an FHOG feature (Felzenszwalb et al., 2010) based correlation filter is learned to estimate the target rotation in each frame.

• We perform an extensive quantitative and qualitative analysis of our tracker and show competitive results on Object Tracking Benchmark (OTB100) (Wu et al., 2015), Visual Object Tracking (VOT) - 2016 Dataset (Kristan et al., 2016d), VOT-2017 Dataset (Kristan et al., 2017b), Tracking Dataset (Vojir et al., 2014) and UAV123 Dataset (Mueller et al., 2016). We also conduct an ablation study to investigate the contribution of various components of our tracker in the tracking performance, demonstrating the benefits of the proposed approach.

The rest of this paper is structured as follows. Section 2 outlines related work. Section 3 describes the proposed tracker with each module in Fig. 1 explained in detail. Section 4 shows evaluation of the LSTM component. Section 5 shows the experimental results obtained by testing the tracker on standard video datasets. It also shows the comparative analysis of the tracker with other recent trackers. Section 6 presents an ablation study that shows the contribution of each module to the overall tracker performance and Section 7 concludes the paper.