NM-GAN: Noise-modulated generative adversarial network for video anomaly detection

Highlights

• A more accurate and stable model for video anomaly detection is achieved within a refined end-to-end GAN-like architecture.

• The reconstruction network has stronger and more controllable generalization ability.

• The discrimination network uses the reconstruction error map to distinguish anomaly samples.

• The proposed noise-modulated adversarial learning method enhances the ability of the discriminator to detect anomalies.

Abstract

As an important and challenging task for intelligent video surveillance systems, video anomaly detection is generally referred to as automatic recognition of video frames that contain abnormal targets, behavior or events. Although it has been widely applied in real scenes, anomaly detection remains a challenging task because of the vague definition of anomaly and the lack of the anomaly samples. Inspired by the widespread application of Generative Adversarial Network (GAN), we propose an end-to-end pipeline called NM-GAN which assembles an encode-decoder reconstruction network and a CNN-based discrimination network in a GAN-like architecture. The generalization ability of the reconstruction network is properly modulated via the adversarial learning around reconstruction error maps and noise maps. Meanwhile, the discrimination network is trained to distinguish anomaly samples from normal samples based on the reconstruction error maps. Finally, the output of the discrimination network is transferred to evaluate anomaly score of the input frame. The thorough proof-of-principle experiments and ablation tests on several popular datasets reveal that the proposed model enhance the generalization ability of the reconstruction network and the distinguishability of the discrimination network significantly. The comparison with the state-of-the-art shows that the proposed NM-GAN model outperforms most competing models in precision and stability.

Introduction

With the rapid development and wide use of video surveillance systems, the conventional manual analysis for labelling anomalies (such as traffic accident, robbery, violent fight, etc.) in the amount of video data captured from public place monitoring is costly, which has not meet the actual requirements of video surveillance systems. Therefore, an intelligent surveillance system that can recognize and detect anomalies is urgently needed and has been a hotspot of the cross field of computer vision and pattern recognition in recent years [1], [2].

Video anomaly detection is a task of finding and identifying anomalous targets, behaviors, and events in video data. It describes and quantifies all kinds of abnormal events according to a unified criterion rather than using specific semantic labels for fine-grained classification. The most challenging issue in video anomaly detection is that the definition of anomaly is indefinite. In general, “anomaly” refers to the observation results that do occur infrequently and do not conform to expectations, or are significantly different from most objects in a scene, which means anomalies are defined by normal events instead of classifications or details of themselves. During the study, people usually assume that samples that frequently appeared in the dataset are normal, while samples that rarely observed in the dataset are anomalies. The training set of existing anomaly detection dataset contains only normal event samples while the testing set contains normal event samples and abnormal event samples. Therefore, the essence of anomaly detection is novelty detection, which is also called one-class learning. Video anomaly detection can be applied to intelligent surveillance system [3], defect detection [4], medical image processing [5], fault diagnosis [6], and so on.

Although the early anomaly detection methods mostly use the frameworks of manual features matching with domain knowledge, with the rapid development of deep learning, the end-to-end frameworks based on deep neural network, including many different methods like probability estimation [7], one-class learning [8], [9], frame reconstruction and prediction [10], [11], adversarial learning [8], [12] and so on, have gradually become dominant because of their advantages in detection speed, robustness and accuracy. Among all these methods, the frame-reconstruction approach received broad attention. A typical reconstruction-based anomaly detection model generally supposes that the anomaly samples cannot be effectively recovered by the reconstruction network if it is trained only with the normal samples. Therefore, the reconstruction error is used to estimate the anomaly of the new samples in these methods. A satisfactory result is obtained in [13], which uses an Auto-encoder network to reconstruct frame sequence and calculate the mean squared error of reconstruction for anomaly detection. A two-branch model [14] combining reconstruction and prediction by 3D convolution network is proposed to enhance the learning of motion information of samples. Some works introduce adversarial learning into reconstruction methods. In [15] the authors describe a framework of adversarial learning based on an Auto-encoder, which detect anomaly by combining appearance and motion features in a two-stream structure. In contrast, in [11] the authors use the error between the prediction frame, which originates from maximizing the expectation of the next frame by prior knowledge of the past frames, and the ground truth to quantify the anomaly of the video frame. A novel adversarial auto-encoder [16] within an encoder-decoder-encoder pipeline is designed to capture the training data distribution within both image and latent vector space to implement anomaly detection efficiently. ST-CaAE [17] adopts the cascade structure of a spatial-temporal adversarial auto-encoder and a spatial-temporal convolutional auto-encoder to build a two-stream framework to detect anomalies. Though the reconstruction-based approach is criticized for the uncertainty to the unobserved samples, it can provide the multi-scale representation with the higher spatial resolution for the video frames, and the learning of the reconstruction network is generally independent of any prior knowledge and class labels, which makes it more conducive to achieve the real applications. Therefore, the proposed model is carried out in the reconstruction-based approach.

In order to improve the performance of the anomaly detection algorithms further, we propose an end-to-end anomaly detection framework NM-GAN which combines the reconstruction approach [18], [19] and the GAN-based approach [8] together. Firstly, as shown in Fig. 1, all the normal samples with the addictive white noise are pushed into the reconstruction network which are expected to make the corresponding reconstruction error maps follow a pre-defined normal distribution. Conversely, the reconstruction error maps will deviate from the pre-defined normal distribution if the input is anomaly samples. The difference of the reconstruction error maps between the normal samples and the anomaly samples will be discovered by a discrimination network to detect the anomaly frames.

In summary, the main contributions of this paper are as follows: (1) The task of video anomaly detection is realized well by distinguishing the reconstruction error map of the input frame with a CNN-based discrimination network. (2) A new scheme of noise-modulated adversarial learning is proposed to improve the generalization ability of the reconstruction network as well as the distinguishability of the discrimination network. Our model learns in an end-to-end fashion and achieves the state-of-the-art performance in multiple challenging datasets for video anomaly detection.

The rest of the paper is organized as follows. In Section 2, we discuss the related work of reconstruction-based model and adversarial learning model in anomaly detection. We introduce the overall structure of NM-GAN in Section 3. Then, the process of optimizations and verifications for the anomaly detection approach are implemented through a series of experiments in Section 4. We conclude the paper with a summary, limitations, and future study in Section 5.