Elsevier

Information Sciences

Volume 533, September 2020, Pages 108-119
Information Sciences

Associative memory optimized method on deep neural networks for image classification

https://doi.org/10.1016/j.ins.2020.05.038Get rights and content

Highlights

Abstract

Deep neural networks have achieved excellent performance in the field of image classification. However, even to state-of-the-art deep neural networks, there are still many critical images that are difficult to be classified effectively. Enlightened by the brain function of associative memory, we propose a novel classification optimization method based on deep neural networks to improve image classifiers. Psychologists have studied associative memory for a long time. A popular theory is that ideas and memory are associated together in the mind through experience. By applying this theory to object recognition, our method focuses on using the association among different images of the same category to improve image classifiers based on deep neural networks. The association, which is memorized by the LSTM network in our method, could infer a sequence of associative images and form inner data augmentation effectively. Further, we introduce the LSTM network into an end-to-end deep learning framework to boost the performance of image classifiers. Experiments on four benchmark datasets reveal that our method produces a consistent improvement to the existing powerful classifiers. Moreover, as far as we know, our method achieves the best classification accuracies of 97.3%, 96.0%, and 89.1% on Flower102, Caltech101, and Caltech256 datasets, respectively.

Introduction

Models based on Convolutional Neural Networks (CNNs) have dominated image classification tasks recently, such as VGG [32], InceptionNet [34], ResNet [15], and DenseNet [20]. They strive to extract powerful deep features to represent images and simultaneously increase the inter-class difference, so as to achieve better classification performance [27]. However, the performance of CNN classifiers is often limited by the image distribution due to the diversity and complexity of images. It is difficult to solve the above problem only by optimizing the internal structure of CNNs. A viable way for this problem is to mimic the cognition process of human beings to reduce data dependence, detect kernel features of images and rely more on reasoning.

The good performance of human cognition on complex object recognition tasks is contributed by the reasoning ability that is based on the brain function of association memory. A popular theory of association memory is that ideas and memory are associated together in the mind through experience [3]. When people recognize objects, associative memory could help to recall the relations among different items, such as a face and a name, to produce new associated items [28]. The study of associative memory is of great significance to the development of artificial intelligence. Researches have obtained significant progress on the association memory. For example, Hopfield neural network [17], a typical representative of the associative memory network and recurrent neural network, can be applied to restore unclear and blurred images. Nowadays, spiking neural network is also researched deeply, and it is used to construct an associative memory system, which reveals that neural networks have powerful association memory ability [14]. Moreover, there are many studies that focus on the association among entities, such as WordNet [29] and Knowledge Graph [33], which explore the association of semantic information. By exploring the associative memory function of LSTM [16], [18] proposes a novel deep network equipped with recurrently aggregated deep features (RADF) to detect salient objects from an image. However, there are few kinds of research on the application of associative memory for image classification.

We propose an Associative Memory Optimized Method on deep neural networks for Image Classification (AMOC), which enhances the performance of the existing convolutional neural networks by introducing the association among images. Firstly, we aggregate the training images into several clusters to establish the association between each image and its representative central image. Secondly, we combine an LSTM-based association memory model with the classifier that needs to be optimized, and the whole network is trained in an end-to-end manner by optimizing a hybrid objective function. In this way, not only the association memory model could generate more identifiable samples, but also the classifier could obtain higher robustness. The theory of association memory in our method is illustrated in Fig. 1. As shown in Fig. 1, samples are distributed in a multi-dimensional space, and classifiers want to find better hyperplanes to separate these samples, while the performance of classifiers is limited by the practical data distribution. Normally, the area near the hyperplane contains most of the error-prone samples. To address this problem, AMOC generates new samples according to the existing association memory, which augments training data internally and reduces those error-prone samples. In this way, the classifier that needs to be optimized could break through the limit of data distribution easily and achieves better performance. AMOC is evaluated on four famous datasets (i.e. MNIST [24], Flower102 [31], Caltech101 [9], and Caltech256 [13]). We provide the experimental results of using and not using AMOC on several advanced classifiers. The visualization results on the MNIST dataset qualitatively demonstrate how the AMOC works, and the experiments on other datasets quantitatively show the effectiveness of our method. Finally, we conduct ablation studies to analyze the factors that affect the performance of AMOC. Experimental results on the four datasets show that AMOC is an effective optimization method, which can optimize the existing advanced classifiers consistently.

The main contributions of this paper can be summarized as follows: (1) We propose an associative memory optimized method on deep neural networks for image classification, which promotes more accurate classification by learning the associations among images. (2) Experiments on four datasets demonstrate that our method is reasonable and effective, and it achieves state-of-the-art performance on Flower102, Caltech101, and Caltech256 datasets. (3) We reveal that association memory is an important process in human cognition, which could be widely used in deep learning to improve the intelligence of deep neural networks.

Section snippets

Related work

Convolutional Neural Networks (CNNs) perform well in image classification. VGG networks [32] are constructed by repeatedly stacking 3×3 small convolution kernels and 2×2 max-pooling layers, which explore the relationship between the depth of a convolutional neural network and its performance. Inception networks [34] contain multiple convolution operations and pooling operations on input images in parallel, and they stitch all output results into a very deep feature map. ResNet [15] and DenseNet

Associative memory optimized method on deep neural networks

By applying the associative memory theory to image classification, we propose an associative memory optimized method to improve the performance of deep neural networks. The associative memory optimized method, denoted as AMOC, relies on the exploration of recurrent neural networks (RNNs), especially the LSTM which has a powerful memory for the association. It is generally known that RNNs could learn the associations among objects, and could associate a new object according to the existing

Experiments

We evaluate AMOC on MNIST, Flower102, Caltech101 and Caltech256 datasets. The visualization results on MNIST reveal how AMOC works, and the results on the other datasets demonstrate that AMOC is effective and efficient. Furthermore, we discuss the memorization of different associative models and demonstrate the robustness of our method by analyzing some reasonable results. Finally, we explore the factors that contribute to the performance of AMOC.

Conclusion

We propose an associative memory optimized method on deep neural networks to improve the classifiers for image classification. Our method consists of two parts, i.e., establishing associations among images and learning these associations. We use clustering algorithms to find several prototype samples in each category and establish associations between samples and their corresponding prototype samples. Moreover, we propose a hybrid CNN-LSTM architecture to learn the established associations,

References (42)

  • J. Donahue et al.

    Long-term recurrent convolutional networks for visual recognition and description

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2017)
  • K. Dutta, P. Krishnan, M. Mathew, C.V. Jawahar, Improving cnn-rnn hybrid networks for handwriting recognition, in: 2018...
  • L. Fei-Fei et al.

    One-shot learning of object categories

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2006)
  • H. Fukui et al.

    Attention branch network: learning of attention mechanism for visual explanation

  • X. Glorot et al.

    Deep sparse rectifier neural networks

  • A. Graves, Generating sequences with recurrent neural networks. Comput. Sci....
  • G. Griffin et al.

    Caltech-256 object category dataset

    (2007)
  • H. He, Y. Shang, X. Yang, Y. Di, J. Lin, Y. Zhu, W. Zheng, J. Zhao, M. Ji, L. Dong, et al., Constructing an associative...
  • K. He et al.

    Deep residual learning for image recognition

  • S. Hochreiter et al.

    Long short-term memory

    Neural Comput.

    (1997)
  • J.J. Hopfield

    Neural networks and physical systems with emergent collective computational abilities

    Proc. Natl. Acad. Sci.

    (1982)
  • Cited by (26)

    • A novel un-supervised burst time dependent plasticity learning approach for biologically pattern recognition networks

      2023, Information Sciences
      Citation Excerpt :

      It is difficult to endow spiking networks with learning methods that yield high accuracy and convergence speed due to their dynamic and complex nature [17]. For this reason, in recent years, deep networks were mapped onto spiking networks and spike-based back-propagation used for supervised learning [6,18–20]. One of the advantages of the presented learning method and topology, when compared to previous spiking platforms, is that it reaches a superior classification accuracy and fastest convergence speed for MNIST, EMNIST and CIFAR10 datasets without the need for the said mapping and supervised learning.

    • Balance optimization method of energy shipping based on Hopfield neural network

      2023, Alexandria Engineering Journal
      Citation Excerpt :

      Its role has not only been paid attention by the government, but also its optimization has attracted more and more attention from energy shipping and energy shipping, including energy shipping and energy shipping and research units. Therefore, the research on transportation issues is very meaningful [8–11]. Using the ergodic property and quasi-randomness of chaos and other rich dynamic characteristics, global ergodic search can improve Hopfield's defects and make the network jump out of the local minimum to achieve global optimization.

    • Mission reconstruction for launch vehicles under thrust drop faults based on deep neural networks with asymmetric loss functions

      2022, Aerospace Science and Technology
      Citation Excerpt :

      Due to the implementation of the interaction process, the computational burden of reinforcement learning tends to raise the threat level of practical application [24]. Recently, deep neural network (DNN) has been applied in many state-of-the-art fields of image, object detection, natural language processing, and other engineering domain [25–27]. DNN-based trajectory planning has applications in the trajectory design of orbit transfer [28–30], trajectory planning for irregular asteroid landings [31], and trajectory planning for a free-floating space robot [32].

    View all citing articles on Scopus

    This work is supported by the Beijing Natural Science Foundation (No. 4192029), the National Natural Science Foundation of China (Nos. 61773385, 61672523).

    View full text