Self-attention binary neural tree for video summarization

Highlights

• A self-attention binary neural tree (SABTNet) is proposed for video summarization.

• SABTNet is the first attempt to combine self-attention modules and decision trees for video summarization.

• SABTNet follows a divide-and-conquer way to evaluate a video shot.

• SABTNet takes hierarchical coarse-to-fine video shot feature learning by self-attention.

• Experimental results show that the proposed SABTNet achieves state-of-the-art performance.

Abstract

In this paper, we address the problem of shot-level video summarization, which aims at selecting a subset of video shots as a summary to represent the original video contents compactly and completely. Most existing methods rely on various network architectures to learn a single score predictor for shot ranking and selection. Different from these methods, we plug network feature learning into a binary neural tree to consider multi-path predictions for each shot, thus enabling the shot evaluation from different aspects. Due to the hierarchical structure of the tree, video shots can be coarse-to-fine encoded by imposing self-attention on them along branches, leading to favorable predictions. Extensive experiments were conducted on two real-world datasets, and the results reveal that the proposed method achieves superior performance in comparison with previous state-of-the-art methods.

Introduction

In recent years, video summarization has become a hot research topic, especially when the explosive increase of videos uploaded to video-sharing platforms by personal users. Because of the lack of professional shooting and editing skills, most videos are usually redundant and time-consuming for browsing. In such a case, one may need video summarization to generate a compact and non-redundant summary for a given video while preserving its main information. This technology can be applied to a lot of real-world domains, including movie recapping, surveillance video browsing, and sports video highlighting. The methods in video summarization can be at object-level [1], [2], frame-level [3], or shot-level [4], according to the visual primitives (objects, frames, or shots) we are processing.

Frame-level and shot-level methods are most commonly investigated in video summarization. Modern frame-level methods [3], [5], [6] usually rely on Long Short-Term Memory (LSTM) to capture long-term and short-term dependencies within a video, and on carefully designed frame scoring networks to predict the probability of each frame being selected into the video summary. For example, an LSTM followed by a fully connected network is adopted to predict the frame scores in Zhang et al. [5]. Recently, LSTM has also been combined with generative adversarial learning [7] and reinforcement learning [6] for frame selection training. Although frame-level methods obtain some promising results, there are two big challenges for these methods: (i) Adjacent frames are very similar in vision, which makes it difficult for the frame scoring networks to have an accurate selection; (ii) the widely used LSTM networks in frame-level methods are not capable enough to explore long-term dependencies of more than 100 frames [4]. To address these two challenges, a frequently-used solution is hierarchical modeling inspired by the hierarchical structure of video-shots-frames, giving rise to the research of shot-level video summarization. By considering video summarization as a hierarchical modeling problem, one can use current shot-level methods [4], [8], [9] to encode frames within each continuous shot into shot-level features, and predict the selection scores on segmented shots rather than on separate frames, which facilitates exploiting temporal similarities and dependencies within a video. In this paper, we also focus on shot-level video summarization. A shot we consider here is a clip of the original video, consisting of a sequence of consecutive frames that are highly cohesive. Different from previous methods that directly learn a single score predictor for frame/shot evaluation and ranking [4], [5] depending on individual feature learning architectures, we enable evaluating a shot from different aspects by a tree structure. As shown in Fig. 1, people may ease to follow a divide-and-conquer way to evaluate a shot from diversity, representativeness, and other factors. Motivated by this intuition, we view shot-level video summarization as a regression decision tree scoring problem, where different root-to-leaf paths indicate different evaluation factors.

Inspired by Tanno et al. [10] and Ji et al. [11], we design a self-attention binary neural tree (SABTNet) for video summarization. Specifically, we start with dividing a video into several shots and adopt a bi-directional Long Short-Term Memory (Bi-LSTM) to extract shot-level visual features. We also incorporate self-attention along edges of the tree structure to focus on different parts of a video. For the non-leaf nodes, we adopt neural networks as the routing functions to determine the root-to-leaf computational paths. Each leaf node is designed to accompany a score predictor, which represents an assessment function of a factor for shot evaluation. This design combines neural networks and decision trees to gain the benefits of both approaches. It not only possesses the sequence modeling ability of self-attention and nonlinear fitting ability of neural networks, but also inherits the coarse-to-ne hierarchical feature learning ability of decision tree. By this means, different root-to-leaf paths in the SABTNet suggest different evaluation factors. The final prediction score of each video shot can be obtained by a weighted summation of the predictions from all the leaf nodes in the tree.

We follow ACNet [11] to use a fixed height full binary tree as the decision tree structure and adopt the soft decision scheme [12]. We conduct extensive evaluations on two widely used datasets, i.e., SumMe and TVSum. The results demonstrate the superiority of our proposed method compared with previous state-of-the-art methods. We conduct several ablation studies to comprehensively understand the influence of some important parameters. The ablation studies include the heights of the tree, the number of heads in self-attention, the number of stacked transformer blocks, and the summary rate.

The main contributions of our proposed method are summarized as follows:

• We propose a novel yet simple self-attention binary neural tree for shot-level video summarization. This model follows a divide-and-conquer way to evaluate a shot from different aspects by taking a coarse-to-fine manner of shot feature learning.

• To the best of our knowledge, this work is the first attempt to combine self-attention and decision trees for video summarization.

• We conduct various experiments on two benchmark datasets, SumMe [3] and TVSum [13]. The results show that the presented method achieves state-of-the-art performance.