Associative memory optimized method on deep neural networks for image classification☆
Graphical abstract
Introduction
Models based on Convolutional Neural Networks (CNNs) have dominated image classification tasks recently, such as VGG [32], InceptionNet [34], ResNet [15], and DenseNet [20]. They strive to extract powerful deep features to represent images and simultaneously increase the inter-class difference, so as to achieve better classification performance [27]. However, the performance of CNN classifiers is often limited by the image distribution due to the diversity and complexity of images. It is difficult to solve the above problem only by optimizing the internal structure of CNNs. A viable way for this problem is to mimic the cognition process of human beings to reduce data dependence, detect kernel features of images and rely more on reasoning.
The good performance of human cognition on complex object recognition tasks is contributed by the reasoning ability that is based on the brain function of association memory. A popular theory of association memory is that ideas and memory are associated together in the mind through experience [3]. When people recognize objects, associative memory could help to recall the relations among different items, such as a face and a name, to produce new associated items [28]. The study of associative memory is of great significance to the development of artificial intelligence. Researches have obtained significant progress on the association memory. For example, Hopfield neural network [17], a typical representative of the associative memory network and recurrent neural network, can be applied to restore unclear and blurred images. Nowadays, spiking neural network is also researched deeply, and it is used to construct an associative memory system, which reveals that neural networks have powerful association memory ability [14]. Moreover, there are many studies that focus on the association among entities, such as WordNet [29] and Knowledge Graph [33], which explore the association of semantic information. By exploring the associative memory function of LSTM [16], [18] proposes a novel deep network equipped with recurrently aggregated deep features (RADF) to detect salient objects from an image. However, there are few kinds of research on the application of associative memory for image classification.
We propose an Associative Memory Optimized Method on deep neural networks for Image Classification (AMOC), which enhances the performance of the existing convolutional neural networks by introducing the association among images. Firstly, we aggregate the training images into several clusters to establish the association between each image and its representative central image. Secondly, we combine an LSTM-based association memory model with the classifier that needs to be optimized, and the whole network is trained in an end-to-end manner by optimizing a hybrid objective function. In this way, not only the association memory model could generate more identifiable samples, but also the classifier could obtain higher robustness. The theory of association memory in our method is illustrated in Fig. 1. As shown in Fig. 1, samples are distributed in a multi-dimensional space, and classifiers want to find better hyperplanes to separate these samples, while the performance of classifiers is limited by the practical data distribution. Normally, the area near the hyperplane contains most of the error-prone samples. To address this problem, AMOC generates new samples according to the existing association memory, which augments training data internally and reduces those error-prone samples. In this way, the classifier that needs to be optimized could break through the limit of data distribution easily and achieves better performance. AMOC is evaluated on four famous datasets (i.e. MNIST [24], Flower102 [31], Caltech101 [9], and Caltech256 [13]). We provide the experimental results of using and not using AMOC on several advanced classifiers. The visualization results on the MNIST dataset qualitatively demonstrate how the AMOC works, and the experiments on other datasets quantitatively show the effectiveness of our method. Finally, we conduct ablation studies to analyze the factors that affect the performance of AMOC. Experimental results on the four datasets show that AMOC is an effective optimization method, which can optimize the existing advanced classifiers consistently.
The main contributions of this paper can be summarized as follows: (1) We propose an associative memory optimized method on deep neural networks for image classification, which promotes more accurate classification by learning the associations among images. (2) Experiments on four datasets demonstrate that our method is reasonable and effective, and it achieves state-of-the-art performance on Flower102, Caltech101, and Caltech256 datasets. (3) We reveal that association memory is an important process in human cognition, which could be widely used in deep learning to improve the intelligence of deep neural networks.
Section snippets
Related work
Convolutional Neural Networks (CNNs) perform well in image classification. VGG networks [32] are constructed by repeatedly stacking small convolution kernels and max-pooling layers, which explore the relationship between the depth of a convolutional neural network and its performance. Inception networks [34] contain multiple convolution operations and pooling operations on input images in parallel, and they stitch all output results into a very deep feature map. ResNet [15] and DenseNet
Associative memory optimized method on deep neural networks
By applying the associative memory theory to image classification, we propose an associative memory optimized method to improve the performance of deep neural networks. The associative memory optimized method, denoted as AMOC, relies on the exploration of recurrent neural networks (RNNs), especially the LSTM which has a powerful memory for the association. It is generally known that RNNs could learn the associations among objects, and could associate a new object according to the existing
Experiments
We evaluate AMOC on MNIST, Flower102, Caltech101 and Caltech256 datasets. The visualization results on MNIST reveal how AMOC works, and the results on the other datasets demonstrate that AMOC is effective and efficient. Furthermore, we discuss the memorization of different associative models and demonstrate the robustness of our method by analyzing some reasonable results. Finally, we explore the factors that contribute to the performance of AMOC.
Conclusion
We propose an associative memory optimized method on deep neural networks to improve the classifiers for image classification. Our method consists of two parts, i.e., establishing associations among images and learning these associations. We use clustering algorithms to find several prototype samples in each category and establish associations between samples and their corresponding prototype samples. Moreover, we propose a hybrid CNN-LSTM architecture to learn the established associations,
References (42)
- et al.
Recurrently exploring class-wise attention in a hybrid convolutional and bidirectional lstm network for multi-label aerial image classification
ISPRS J. Photogrammetry Remote Sens.
(2019) - et al.
Videolstm convolves, attends and flows for action recognition
Comput. Vis. Image Underst.
(2018) - et al.
Effects of non-invasive brain stimulation on associative memory
Brain Res.
(2015) - et al.
Ta-cnn: Two-way attention models in deep convolutional neural network for plant recognition
Neurocomputing
(2019) - et al.
Image classification using a hybrid lstm-cnn deep neural network
Int. J. Eng. Adv. Technol. (IJEAT)
(2019) - et al.
Fusion of transfer learning features and its application in image classification
- et al.
Human associative memory
(2014) - et al.
Photographic image synthesis with cascaded refinement networks
IEEE International Conference on Computer Vision (ICCV)
(2017) - et al.
Empirical evaluation of gated recurrent neural networks on sequence modeling
(2014) - I. Danihelka, G. Wayne, B. Uria, N. Kalchbrenner, A. Graves, Associative long short-term memory, ICML,...
Long-term recurrent convolutional networks for visual recognition and description
IEEE Trans. Pattern Anal. Mach. Intell.
One-shot learning of object categories
IEEE Trans. Pattern Anal. Mach. Intell.
Attention branch network: learning of attention mechanism for visual explanation
Deep sparse rectifier neural networks
Caltech-256 object category dataset
Deep residual learning for image recognition
Long short-term memory
Neural Comput.
Neural networks and physical systems with emergent collective computational abilities
Proc. Natl. Acad. Sci.
Cited by (26)
ResfEANet: ResNet-fused External Attention Network for Tuberculosis Diagnosis using Chest X-ray Images
2024, Computer Methods and Programs in Biomedicine UpdateGeneral decay anti-synchronization and H<inf>∞</inf> anti-synchronization of coupled reaction-diffusion complex-valued memristive neural networks
2023, Communications in Nonlinear Science and Numerical SimulationA novel un-supervised burst time dependent plasticity learning approach for biologically pattern recognition networks
2023, Information SciencesCitation Excerpt :It is difficult to endow spiking networks with learning methods that yield high accuracy and convergence speed due to their dynamic and complex nature [17]. For this reason, in recent years, deep networks were mapped onto spiking networks and spike-based back-propagation used for supervised learning [6,18–20]. One of the advantages of the presented learning method and topology, when compared to previous spiking platforms, is that it reaches a superior classification accuracy and fastest convergence speed for MNIST, EMNIST and CIFAR10 datasets without the need for the said mapping and supervised learning.
Balance optimization method of energy shipping based on Hopfield neural network
2023, Alexandria Engineering JournalCitation Excerpt :Its role has not only been paid attention by the government, but also its optimization has attracted more and more attention from energy shipping and energy shipping, including energy shipping and energy shipping and research units. Therefore, the research on transportation issues is very meaningful [8–11]. Using the ergodic property and quasi-randomness of chaos and other rich dynamic characteristics, global ergodic search can improve Hopfield's defects and make the network jump out of the local minimum to achieve global optimization.
Class-specific weighted broad learning system for imbalanced heartbeat classification
2022, Information SciencesMission reconstruction for launch vehicles under thrust drop faults based on deep neural networks with asymmetric loss functions
2022, Aerospace Science and TechnologyCitation Excerpt :Due to the implementation of the interaction process, the computational burden of reinforcement learning tends to raise the threat level of practical application [24]. Recently, deep neural network (DNN) has been applied in many state-of-the-art fields of image, object detection, natural language processing, and other engineering domain [25–27]. DNN-based trajectory planning has applications in the trajectory design of orbit transfer [28–30], trajectory planning for irregular asteroid landings [31], and trajectory planning for a free-floating space robot [32].
- ☆
This work is supported by the Beijing Natural Science Foundation (No. 4192029), the National Natural Science Foundation of China (Nos. 61773385, 61672523).