Latent Dirichlet allocation based generative adversarial networks

Abstract

Generative adversarial networks have been extensively studied in recent years and powered a wide range of applications, ranging from image generation, image-to-image translation, to text-to-image generation, and visual recognition. These methods typically model the mapping from latent space to image with single or multiple generators. However, they have obvious drawbacks: (i) ignoring the multi-modal structure of images, and (ii) lacking model interpretability. Importantly, the existing methods mostly assume one or more generators can cover all image modes even if we do not know the structure of data. Thus, mode dropping and collapse often take place along with GANs training. Despite the importance of exploring the data structure in generation, it has been almost unexplored. In this work, aiming at generating multi-modal images and interpreting model explicitly, we explore the theory on how to integrate GANs with data structure prior, and propose latent Dirichlet allocation based generative adversarial networks (LDAGAN). This framework is extended to combine with a variety of state-of-the-art single-generator GANs and achieves improved performance. Extensive experiments on synthetic and real datasets demonstrate the efficacy of LDAGAN for multi-modal image generation. An implementation of LDAGAN is available at https://github.com/Sumching/LDAGAN.

Introduction

Generating realistic images has been actively pursued in both machine learning and computer vision communities in recent years. Generative adversarial networks (GANs) (Arjovsky et al., 2015, Berthelot et al., 2017, Goodfellow et al., 2014, Hoang et al., 2018, Mao et al., 2017, Miyato et al., 2018, Zhang et al., 2019) provide us a promising way to achieve this goal and its remarkable ability has powered a wide range of applications, ranging from image generation (Gulrajani et al., 2017, Yang et al., 2018), image-to-image translation (Liu et al., 2017, Tran et al., 2017, Yi et al., 2017, Zhu et al., 2017), to text-to-image generation (Reed et al., 2016, Zhang et al., 2017), and visual recognition (Li et al., 2017). Generally, GANs are required to generate output to fit the multi-modal distribution of real images.

However, this goal is hard to achieve if the underlying structure of data is unclear (Pan et al., 2016). This partially explains why mode collapse and dropping problems take place along with GANs training (Arjovsky et al., 2015, Heusel et al., 2017). Typically, employing multiple generators $G_k,k=1,\dots ,K$, could generate more diverse and varied images (Arora et al., 2017, Hoang et al., 2018, Tolstikhin et al., 2017), where the generative distribution $p_g\left(\mathbf{x}\right)$ is an assemble of sub-modal distributions $p_{g_k}\left(\mathbf{x}\right),k=1,\dots ,K$, that is, $p_g\left(\mathbf{x}\right)={\sum }_{k=1}^K{\pi }_k{p}_{g_k}\left(\mathbf{x}\right)$. However, they still ignore the underlying structure of images, resulting in problems like mode dropping. The absence of sample likelihood and posterior of latent variables in GANs seems to prohibit the forward step to fix this drawback. Moreover, these approaches lack model interpretation: how multiple generators correlate to data structure.

In this paper, we construct a probabilistic GAN framework for multi-modal image generation, which not only integrates with data structure prior, but also has explicit model interpretation (see Fig. 1). We motivate our construction through a Bayesian network. In the following, we list the three main contributions.

First, we introduce the Dirichlet prior and frame a new probabilistic GAN: latent Dirichlet allocation based GAN (LDAGAN). (i) Unlike existing GANs which model the latent space with a simple unimodal distributions (Berthelot et al., 2017, Goodfellow et al., 2014, Gulrajani et al., 2017, Miyato et al., 2018, Peng et al., 2019), we define additional discrete variables in latent space to represent the structure (i.e. mode) of data. Moreover, we impose a hierarchical prior on structure, which increases the possible solution spaces for the multi-modal structure prior (see Fig. 2(b)). (ii) Depending on data structure prior, we build structured GAN to precisely fit complex image data—each generator is only responsible for one image mode. (iii) Through Bayesian networks, we explicitly interpret how multiple generators correlate to data structure.

Second, since GANs do not have an explicit sample likelihood, a natural question then arises: how to estimate model parameters and posterior distributions of latent variables in LDAGAN? To this end, we take an important step: (i) We formulate the likelihood function for LDAGAN model by virtue of the discriminator in GANs. (ii) With the above likelihood, we present a variational inference algorithm in GANs and solve for model parameters by virtue of EM algorithm. (iii) We make stochastic variation inference (Hoffman et al., 2013) to ensure the training of LDAGAN is not time-consuming.

Third, we extend the LDAGAN framework to combine with other single-generator GANs, such as SNGAN (Miyato et al., 2018) and VGAN (Peng et al., 2019). We achieve state-of-the-art performance on both CIFAR-10 (Krizhevsky & Hinton, 2009) and CIFAR-100 (Krizhevsky & Hinton, 2009) datasets. For example, our method has achieved a value of 10.4 for Fréchet Inception Distance on the CIFAR-10 dataset, which is currently the best reported result with ResNet architecture in literature to our knowledge.

The theoretical analysis of this paper may further inspire the GAN communities to study and improve upon how to integrate GANs with other prior knowledge, pursuing more realistic generative performance and explicit interpretability, since this problem thus far has not been well explored.