Elsevier

Pattern Recognition

Volume 107, November 2020, 107440
Pattern Recognition

Generative attention adversarial classification network for unsupervised domain adaptation

https://doi.org/10.1016/j.patcog.2020.107440Get rights and content

Highlights

  • We propose an approach to solve the unsupervised domain adaptation.

  • We provide an improved generative adversarial network following the feature extractor F to learn a joint feature distribution between source and target domains.

  • We present an attention module in the process of adversarial learning, which allows the discriminator to distinguish the transferable regions among the source and target images.

  • We propose the simple and efficient method of giving unlabeled target domain pseudo labels, which helps us obtain a part of the category information of target domain data and can improve the performance of our model and mitigate negative transfer at the same time.

  • Experiments demonstrate that our model achieves excellent result s on several standard domain adaptation datasets.

Abstract

Domain adaptation is a significant and popular issue of solving distribution discrepancy among different domains in computer vision. Generally, previous works proposed are mainly devoted to reducing domain shift between source domain with labeled data and target domain without labels. Adversarial learning in deep networks has already been widely applied to learn disentangled and transferable features between two different domains to minimize domains distribution discrepancy. However, these methods rarely consider class distributions among source data during adversarial learning, and they pay little attention to these transferable regions among source and target domains images. In this paper, we propose a Generative Attention Adversarial Classification Network (GAACN) model for unsupervised domain adaptation. To learn a joint feature distribution between source and target domains, we present an improved generative adversarial network (GAN) following the feature extractor. Firstly, the discriminator of GAN discriminates the distribution of domains and the classes distribution among source data during adversarial learning, so that our feature extractor can learn a joint feature distribution between source and target domains and maintain the classes consistent simultaneously. Secondly, we present an attention module embedded in GAN, which allows the discriminator to discriminate the transferable regions among the images of source and target domains. Lastly, we propose a simple and efficient method which allocates pseudo-labels for unlabeled target data, and it can improve the performance of our model GAACN while mitigating negative transfer. Extensive experiments demonstrate that our proposed model achieves perfect results on several standard domain adaptation datasets.

Introduction

With the help of large-scale labeled data, deep learning technology has greatly improved the performance of diverse applications in many fields, such as computer vision [1], [2], [3], [4] and natural language processing [5], [6], [7]. In real world application scenarios, we can easily obtain a large number of unlabeled samples, which are referred as target data. And we have a labeled source data which has a similar but not the same data distribution to target data. However, owing to a phenomenon known as domain shift, deep networks trained on these labeled source data do not generalize well to these new and unlabeled data or scenarios. Therefore, how to minimize their domain shift between labeled source data and unlabeled target data raises the problem of domain adaptation [8], [9].

In many unsupervised domain adaptation tasks, we need to train a deep neural network that can transfer knowledge from a labeled source domain having sufficient training data to an unlabeled target domain, but this is blocked by data distributions discrepancy among different domains. The simple and effective fine-tune method has improved the performance of transfer learning. However, this prevalent fine-tune method using pre-trained models on task-specific datastes may be not very practical. Because we have to spend too much for collecting enough labeled and correlative data for fine-tuning the considerable number of deep neural network parameters. Further on, this dilemma has promoted the research on domain adaptation, which aims to propose effective algorithms to reduce labeling cost and leverage available labeled source data.

In traditional machine learning methods, some previous methods aimed to solve this problem by estimating sample importance among source domain with labeled data and unlabeled target data [10], [11]. While some approaches focused on bridging source and target domains by learning domain-invariant representations [12], [13]. Recently, many works are devoted to addressing domain adaptation issue by embedding the adaptive modules in deep networks. The main technologies adopted by most researchers for matching data distribution are moment matching method [14], [15], [16] and adversarial learning approach [17], [18], [19]. These approaches [15], [16] on domain adaptation for classification task are mainly to extract domain-invariance representations to minimize the feature distributions discrepancy by domain adaptation modules existing in deep network architectures. Adversarial domain adaptation [17], [19], [20] mainly combines domain adaptation module and adversarial learning which is similar to generative adversarial networks (GANs). The domain discriminator minimizes the binary classification error to distinguish source domain from target domain, so that the model can determine these transferable features which are not distinguishable for domain discriminator.

Although deep Convolution Neural Networks (CNNs) have ability to transfer knowledge from labeled source data to unlabeled target data, the lack of uniformity of data distributions across different domains results in suboptimal performance. These methods [17], [18], [19] do not consider maintaining the classes consistency in the distribution of source data, which may result in missing the important classes information in the process of adversarial learning. And they pay little attention to these transferable regions among source and target domains images. Moreover, the label information of target data is also vital important for CNNs, while they seldom consider these effective information. In this paper, we propose the model of Generative Attention Adversarial Classification Network (GAACN) as shown in Fig. 1 for domain adaptation. In summary, the main contributions in this paper are as follows.

  • We propose an improved generative adversarial network following the feature extractor F to learn a joint feature distribution between source and target domains. The discriminator is able to distinguish the distribution of domains and the classes distribution among source data simultaneously. In this way, our feature extractor F can learn the joint feature distribution and maintain the source domain samples categories consistency during adversarial learning.

  • We present an attention module in the process of adversarial learning, which allows the discriminator to distinguish the transferable regions among the source and target images. Therefore, it contributes to further minimize domain shifts.

  • We are committed on improving the classification capabilities of classifier C. Hence, we propose the simple and efficient method of giving unlabeled target domain pseudo-labels, which not only helps us obtain a part of the category information of target data, but also can improve the performance of our model GAACN and mitigate negative transfer at the same time.

  • Comprehensive experiments demonstrate that our model GAACN achieves excellent performance on several standard domain adaptation datasets.

Section snippets

Related work

Large-scaled annotated data and increasing computational power have enabled the rapid development of deep learning. CNNs have a high performance on many computer vision tasks, such as image classification [21], semantic segmentation [22], [23] and object detection [24], [25]. Although deep learning technology has been greatly developed, it still can not have strong generalization ability to new environments behaving like humans. When the distribution of source and target domains is different,

Proposed method

In this section, we are going to introduce the proposed approach GAACN as shown in Fig. 1. Firstly, let’s introduce some terms and symbols used in our paper. For the issue of unsupervised domain adaptation, we are given a labeled source domain and totally unlabeled target domain. Let xs={xi}i=1ns and xt={xi}i=1nt be the input spaces of source and target respectively, and ys={yi}i=1ns is the label space with labels set Label={1,2,3,,Nc} where Nc is the total number of classes of both source and

Experiments

In this section, we evaluate performance of our proposed model GAACN with five commonly used standard domain adaptation datasets. They are Digits, ImageCLIEF-DA, Office31, Office-Home and visDA-2017. Some samples are shown in Fig. 2. In all our experiments, we follow the standard unsupervised protocol using all labeled source data and the unlabeled data from target domain. Further ablation experiments are also performed to evaluate the effectiveness of our proposed modules.

Conclusion

In this paper, we propose a Generative Attention Adversarial Classification Network (GAACN) model for unsupervised domain adaptation. With regard to the issue of domain adaptation, we are concentrated on the design of the feature extractor F and the classifier C. Firstly, we combine a generative adversarial network (GAN) following F to learn a joint feature distribution between source and target domains, so that F can learn the domain-invariance representations. Moreover, the discriminator D

Declaration of Competing Interest

We wish to confirm that there are no known conflicts of interest associated with this publication and there has been no significant financial support for this work that could have influenced its outcome.

Acknowledgement

This work was supported in part by the National Natural Science Foundation of China (61673402, 61273270 and 60802069), the Natural Science Foundation of Guangdong Province (2017A030311029), and the Science and Technology Program of Guangzhou, China, under Grant 201704020180, and the Fundamental Research Funds for the Central Universities of China.

Wendong Chen received the M.S. degree from School of Electronics and Information Technology at Sun Yat-sen university. Now he is a graduate and majors in information and communication engineering. His research interest is mainly computer vision.

References (67)

  • J. Donahue et al.

    DeCAF: A deep convolutional activation feature for generic visual recognition

  • J. Hoffman et al.

    LSDA: Large scale detection through adaptation

    Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2

    (2014)
  • R. Gopalan et al.

    Domain adaptation for object recognition: an unsupervised approach

    Proceedings of the 2011 International Conference on Computer Vision

    (2011)
  • S. Ruder et al.

    Strong baselines for neural semi-supervised learning under domain shift

    Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    (2018)
  • T. Young et al.

    Recent trends in deep learning based natural language processing

    IEEE Comput. Intell. Mag.

    (2018)
  • S.J. Pan et al.

    A survey on transfer learning

    IEEE Trans. Knowl. Data Eng.

    (2010)
  • W. Dai et al.

    Boosting for transfer learning

    Proceedings of the 24th International Conference on Machine Learning

    (2007)
  • M.N.A. Khan et al.

    Adapting instance weights for unsupervised domain adaptation using quadratic mutual information and subspace learning

    2016 23rd International Conference on Pattern Recognition (ICPR)

    (2016)
  • K.M. Borgwardt et al.

    Integrating structured biological data by kernel maximum mean discrepancy

    (2006)
  • B. Sun et al.

    Return of frustratingly easy domain adaptation

    Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence

    (2016)
  • M. Long et al.

    Learning transferable features with deep adaptation networks

    Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37

    (2015)
  • M. Long et al.

    Deep transfer learning with joint adaptation networks

    Proceedings of the 34th International Conference on Machine Learning - Volume 70

    (2017)
  • M. Long et al.

    Unsupervised domain adaptation with residual transfer networks

    Proceedings of the 30th International Conference on Neural Information Processing Systems

    (2016)
  • Y. Ganin et al.

    Unsupervised domain adaptation by backpropagation

    Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37

    (2015)
  • M. Long et al.

    Deep transfer learning with joint adaptation networks

    Proceedings of the 34th International Conference on Machine Learning - Volume 70

    (2017)
  • E. Tzeng et al.

    Adversarial discriminative domain adaptation

    2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    (2017)
  • S. Sankaranarayanan et al.

    Generate to adapt: aligning domains using generative adversarial networks

    2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

    (2018)
  • K. He et al.

    Deep residual learning for image recognition

    2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    (2016)
  • J. Hoffman et al.

    CyCADA: Cycle-consistent adversarial domain adaptation

  • C. Szegedy et al.

    Deep neural networks for object detection

    Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2

    (2013)
  • E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, T. Darrell, Deep domain confusion: maximizing for domain invariance,...
  • M. Long et al.

    Transferable representation learning with deep adaptation networks

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (2018)
  • W. Zellinger, T. Grubinger, E. Lughofer, T. Natschläger, S. Saminger-Platz, Central moment discrepancy (CMD) for...
  • Cited by (44)

    View all citing articles on Scopus

    Wendong Chen received the M.S. degree from School of Electronics and Information Technology at Sun Yat-sen university. Now he is a graduate and majors in information and communication engineering. His research interest is mainly computer vision.

    Haifeng Hu received the Ph.D. degree from Sun Yat-sen University in 2004, and he is a professor of School of Electronics and Information Technology at Sun Yat-sen University. His research interests are in computer vision, pattern recognition, image processing and neural computation. He has published about 120 papers since 2000.

    View full text