Mimetics: Towards Understanding Human Actions Out of Context

Abstract

Recent methods for video action recognition have reached outstanding performances on existing benchmarks. However, they tend to leverage context such as scenes or objects instead of focusing on understanding the human action itself. For instance, a tennis field leads to the prediction playing tennis irrespectively of the actions performed in the video. In contrast, humans have a more complete understanding of actions and can recognize them without context. The best example of out-of-context actions are mimes, that people can typically recognize despite missing relevant objects and scenes. In this paper, we propose to benchmark action recognition methods in such absence of context and introduce a novel dataset, Mimetics, consisting of mimed actions for a subset of 50 classes from the Kinetics benchmark. Our experiments show that (a) state-of-the-art 3D convolutional neural networks obtain disappointing results on such videos, highlighting the lack of true understanding of the human actions and (b) models leveraging body language via human pose are less prone to context biases. In particular, we show that applying a shallow neural network with a single temporal convolution over body pose features transferred to the action recognition problem performs surprisingly well compared to 3D action recognition methods.

Appendices

Extended Experiments on Existing Datasets

In this section, we provide more analysis about the performance of the three pose-based baselines on existing action recognition datasets. We first perform a parametric study of SIP-Net in “SIP-Net baseline” of appendix. We then use the various levels of ground-truth (see Table 3 of the main paper) to study the impact of using ground-truth or extracted tubes and poses (“Comparison between baselines” of appendix).

Tubes For datasets with ground-truth 2D poses, we compare the performance when using ground-truth tubes (GT Tubes) obtained from GT 2D poses, or estimated tubes (LCR Tubes) built from estimated 2D poses, see Sect. 4.1 of the main paper. In the latter case, tubes are labeled positive if the spatio-temporal IoU with a GT tube is over 0.5, and negative otherwise. When there is no tube annotation, we assume that all tubes are labeled with the video class label. Note that in some videos, no tube is extracted, in which case the videos are ignored when training, and considered as wrongly classified for test videos. In particular, this happens when only the head is visible, as well as for many clips with first person viewpoint, where only one hand or the main manipulated object is visible. We obtain no tube for 0.1% of the videos on PennAction, 2.5% on JHMDB, 2.7% on HMDB51, 6.7% on UCF101 and 15.3% on Kinetics.

1.1 SIP-Net Baseline

We first present the results for the SIP-Net baseline with GT tubes (blue curve ‘GT tubes, Pose Feats’) and LCR tubes (green curve ‘LCR Tubes, Pose Feats’) on all datasets for varying clip length T, see Fig. 7. Overall, a larger clip size T leads to a higher classification accuracy. This is in particular the case for datasets with longer videos such as NTU and Kinetics. This holds both when using GT tubes (blue curve) and LCR tubes (green curve). We keep $T=32$ in the remaining of this paper.

Next, we measure the impact of applying transfer learning from the pose domain to action recognition. To this end, we compare the temporal convolution on LCR pose features (blue curve, ‘Pose Feats’), to features extracted from a Faster R-CNN model with ResNet50 backbone trained to classify actions (red curve, ‘Action Feats’). This latter method is not supposed to be state-of-the-art in action recognition, but it allows to fairly compare the pose features to action features, keeping the network architecture exactly the same, simply changing the learned weights. Note that such a frame-level action detector has been used in the spatio-temporal action detection literature (Saha et al. 2016a; Weinzaepfel et al. 2015b), before the rise of 3D CNNs. Results in Fig. 7 show a clear drop of accuracy when using action features instead of pose features: about 20% on JHMDB-1 and PennAction, and around 5% on NTU for $T=32$. Interestingly, this holds for $T=1$ on HMDB-1 and PennAction, i.e., without temporal integration, showing that ‘Pose feats’ are more powerful. To better understand why using pose features considerably increases performance compared to action features, we visualize the distances between features inside tubes in Fig. 8. When training a per-frame detector specifically for actions, most features of a given tube are correlated. It is therefore hard to leverage temporal information from them. In contrast, LCR-Net++ pose features considerably change over time, as does the pose, deriving greater benefit from temporal integration. Figure 9 shows confusion matrices on PennAction when using ‘Pose feats’ (left) versus ‘Action feats’ (right). With ‘Action feats’, confusions happen between the two tennis or the two baseball actions, while this is disambiguated with ‘Pose feats’.

Fig. 7

Mean-accuracy of SIP-Net for varying T on all datasets, for different tubes (GT or LCR) and features (Pose or Action)

Fig. 8

Feature correlation for two different videos of PennAction (action swing baseball on the left and bench press on the right). For each sequence, we show the distances between features along the tube when using Faster R-CNN action or LCR pose features. Blue/red color indicates low/high distances and therefore high/low correlation. Implicit pose features clearly show more variation inside a tube (Color figure online)

Fig. 9

Confusion matrices on PennAction when using action features (left) and pose features (right) in SIP-Net

Table 10 Mean accuracies (in %) for our three baselines on datasets with GT tubes, or when using LCR tubes

Table 11 Per-class results on the Kinetics for the RGB and Flow 3D CNN baselines, for STGCN (Yan et al. 2018b) (2D with OpenPose), as well as our three baselines (STGCN2D, STGCN3D, SIP-Net)

1.2 Comparison Between Baselines

We compare the performance of the baselines using GT and LCR tubes, on the JHMDB-1, PennAction and NTU datasets in Table 10. On JHMDB-1 and PennAction, despite being a much simpler architecture, the SIP-Net baseline outperforms the methods based on explicit 2D–3D pose representations, both with GT and LCR tubes. Estimated 3D pose sequences are usually noisy and may lack temporal consistency. We also observe that the STGCN3D approach significantly outperforms its 2D counterpart (STGCN2D), confirming that 2D poses contain less discriminative and more ambiguous information.

On the NTU dataset, the 3D pose baseline obtains 74.8% accuracy when using GT tubes and estimated poses (STGCN3D on GT Tubes), compared to 81.5% reported in (Yan et al. 2018b) when using ground-truth 3D poses. This gap of 7% in a constrained environment is likely to increase for videos captured in the wild. The performance of the features-based baseline (SIP-Net) is lower, 66.4% on GT tubes, suggesting than SIP-Net performs better only in unconstrained scenarios.

Per-Class Results on Mimetics

In Table 11, we present for each class the top-1 accuracy and the AP of the different methods. For the top-1 accuracy metric, SIP-Net obtains the best performance for 19 out of 50 classes, with a mean accuracy of 14.2%. The RGB 3D CNN baseline obtains the highest AP for 8 classes, which often correspond to classes in which manipulated objects are small, making the network less bias towards context (e.g. the ball for the action catching of throwing baseball). Table 11 also highlights that the recognition of mimed actions is a very challenging and open task, as none of the videos are correctly classified (i.e. 0% top-1 accuracy) by the 5 baselines for 5 out of the 50 classes.

Mimetics Evaluation with Superclasses

One hypothesis for the low performance of the different methods on Mimetics is the fact that actions are too fine-grained. To evaluate this aspect, we aggregated several actions from the Kinetics dataset into some higher-level classes, denoted as superclasses. Fig. 10 shows the actions that belong to a larger group. Classes not present in one of this group are kept as is, i.e., they form their own superclass.

For evaluation, we still consider models trained on the 400 classes from Kinetics, but at test time, when evaluating on Mimetics, we transformed the 400 class probabilities into superclass probabilities. As a side effect, the number of 50 classes from Mimetics is down to 45 superclasses. Note that some elements in the superclasses do not belong to the subset of classes covered by Mimetics but are still used to sum the probabilities of a superclass from the Kinetics predictions. Results are presented in Table 12 (top part with 50 classes, bottom part with 45 classes). We observe that the performance of all methods slightly increase compared to the evaluation with fine-grained classes (Table 6) but the overall conclusion still holds. Note in particular that performances of pose-based methods show a larger increase, e.g + 10% top-5 accuracy for STGCN3D or + 4% for SIP-Net compared to + 2% for 3D-ResNeXt-101 RGB.

Fig. 10

Superclasses employed in this paper. Classes not mentioned here are kept as is and considered as superclasses

Table 12 Evaluation on Mimetics using superclasses