Complementary Boundary Estimation Network for Temporal Action Proposal Generation

Abstract

Temporal Action Detection is an important yet challenging task, in which temporal action proposal generation plays an important part. Since the temporal boundaries of action instances in videos are often ambiguous, it’s difficult to locate them precisely. Boundary Sensitive Network (BSN) (Lin et al. in ECCV, 2018) is a state-of-the-art corner-based method that can generate high-quality proposals with high recall rate. It contains a temporal evaluation network and a proposal evaluation network to generate and evaluate proposals separately, which can find the temporal boundaries of action instances directly to produce proposals with flexible temporal intervals and evaluate the quality of proposals. But BSN still has some issues: (1) Due to the small reception field of temporal evaluation network, it often generates many false temporal boundaries. (2) Evaluating the quality of proposals is a difficult task and not well solved in the paper. To address these issues, we propose Complementary Boundary Estimation Network (CBEN), an improved approach to temporal action proposal generation based on the framework of BSN. Specifically, we improve BSN in two aspects: Firstly, considering the temporal evaluation network of BSN can only capture local information and tends to have high response at background segments, we combine it with a new network with larger reception field to better identify false temporal action boundaries. Secondly, to evaluate the quality of temporal action proposals more accurately, we propose a class-based proposal evaluation network and combine it with a tIoU-based proposal evaluation network to filter out low-quality proposals. Extensive experiments on THUMOS14 and ActivityNet-1.3 datasets indicate that CBEN can achieve better performance than current mainstream methods on temporal action proposal generation. We further combine CBEN with an off-the-shelf action classifier, and show consistent performance improvements on THUMOS14 dataset.

Appendices

Appendix A: Calculation of tIoU

tIoU is short for temporal Intersection over Union. Figure 13 shows the beginning and ending time of a temporal action proposal and ground truth. We can calculate the tIoU between the proposal and ground truth by

$$\begin{aligned} tIoU=\frac{Intersection}{Union}=\frac{t_2-t_b}{t_e-t_1} \end{aligned}$$

(11)

The higher value of tIoU represents the proposal is closer to ground truth, e.g. the higher quality of the proposal.

Fig. 13

The schematic diagram of ground truth (blue) and proposal (green). $t_b$ and $t_e$ are the beginning and ending time of ground truth, and $t_1$ and $t_2$ are the beginning and ending time of the proposal. (Color figure online)

Appendix B: Derivative of Mean Square Error

Assume we use mean square error to optimize the tIoU-based PEN, then the firs-order derivative of weights in the output layer can be calculated by

$$\begin{aligned} f(\mathbf {w})= & {} sigmoid(\mathbf {w}^\mathrm {T}\mathbf {x}^{(i)})\nonumber \\ J(\mathbf {w})= & {} \frac{1}{2n}\sum _{i=1}^{n}[y_i-f(\mathbf {w})]^2\nonumber \\ \frac{\partial {J}}{\partial {\mathbf {w}}}= & {} -\frac{1}{n}\sum _{i=1}^{n}(f(\mathbf {w})-y_i)(f(\mathbf {w})-1)f(\mathbf {w})\mathbf {x}^{(i)} \end{aligned}$$

(12)

To be noted, $g(x)=(x-y)(x-1)x$ is not an increasing function when $x,y\in (0,1)$, so $\frac{\partial {J}}{\partial {\mathbf {w}}}$ is not a increasing function either which means $J(\mathbf {w})$ is a non-convex function.

Appendix C: Derivative of Softmax Cross Entropy

Assume softmax cross entropy is used to optimize the class-based PEN, then we can calculate the first-order and second-order derivative of weights in the output layer by

$$\begin{aligned} a_j= & {} \frac{e^{\mathbf {w}_j^\mathrm {T}{\mathbf {x}}}}{\sum _{k=1}^{c}e^{\mathbf {w}_k^\mathrm {T}{\mathbf {x}}}}\nonumber \\ J(\mathbf {w})= & {} -\frac{1}{n}\sum _{i=1}^{n}\sum _{j=1}^{c}y_j^{(i)}log a_j^{(i)}\nonumber \\ \frac{\partial {J}}{\partial {\mathbf {w}_n}}= & {} -\frac{1}{n}\sum _{i=1}^{n}[\mathbf {x}^{(i)}(y_n^{(i)}-a_n)]\nonumber \\ \frac{\partial {^2J}}{\partial {\mathbf {w}_n^2}}= & {} \frac{1}{n}\sum _{i=1}^{n}a_n(1-a_n)\mathbf {x}^{(i)}\mathbf {x}^{(i)\mathrm {T}} \end{aligned}$$

(13)

Considering $a_n\in (0,1)$, $\frac{\partial {^2J}}{\partial {\mathbf {w}_n^2}}$ is positive semi-definite matrix, which means $J(\mathbf {w})$ is a convex function.