Training bidirectional generative adversarial networks with hints

Highlights

• The BiGAN has an encoder, in addition to the generator and discriminator of GAN.

• This encoder coupled with the generator allows defining extra loss terms as hints.

• We experiment on five image data sets, MNIST, UT-Zap50K, GTSRB, Cifar10, and CelebA.

• With these different hints, BiGAN generates higher quality and more diverse images.

Abstract

The generative adversarial network (GAN) is composed of a generator and a discriminator where the generator is trained to transform random latent vectors to valid samples from a distribution and the discriminator is trained to separate such “fake” examples from true examples of the distribution, which in turn forces the generator to generate better fakes. The bidirectional GAN (BiGAN) also has an encoder working in the inverse direction of the generator to produce the latent space vector for a given example. This added encoder allows defining auxiliary reconstruction losses as hints for a better generator. On five widely-used data sets, we showed that BiGANs trained with the Wasserstein loss and augmented with hints learn better generators in terms of image generation quality and diversity, as measured numerically by the 1-nearest neighbor test, Fréchet inception distance, and reconstruction error, and qualitatively by visually analyzing the generated samples.

Introduction

In generative modeling, we have a data set of {x^t}_t drawn from an unknown probability distribution p(x) and we would like to be able to generate new x that also look like they have been drawn from p(x). For example, x^t may be the face images of a collection of people and we would like to be able to generate new face images; these new synthetic images would be legitimate faces but of people that do not exist.

The typical approach would be to learn some estimator to p(x) (e.g., using a Gaussian distribution) and then to sample from that estimator. The approach we have in this paper defines generative modeling as a mapping task where a generator function takes some low-dimensional z drawn from a given p(z) as input and transforms it into a valid instance x from p(x). All the structure that p(x) has (for example, all the requirements of being a face image) needs to be captured during learning so that the newly generated x also reflect those.

In many real-world applications that involve, for instance, images, speech, or text, our observations x are high-dimensional; at the same time, we know that all these dimensions are not all necessary or independent. An important research area in machine learning is hence dimensionality reduction where we want to map x to a much lower-dimensional z-space without any loss of information, and many methods, e.g., principal components analysis (PCA), have been proposed to learn such a mapping. In a generative model, we posit that the dimensions of z are latent factors that interact to generate the observed x; one example model is factor analysis (FA), which goes in the opposite direction of PCA.

Unsupervised dimensionality reduction can be learned using the neural network architecture called the autoencoder (AE) (Fig. 1). The encoder part compresses x to z (as in PCA) and the decoder part generates x from z (as in FA). The two networks back-to-back are trained to reconstruct the input, that is, to minimize the difference between the output of the decoder and the input to the encoder. In the simplest case, both the encoder and the decoder are one-layer (i.e., linear) networks and in this case, it has been shown that the encoder spans the same subspace as PCA, but with the encoder and the decoder having more layers, the AE realizes nonlinear dimensionality reduction with z corresponding to more interesting abstract features of the input.

Typically, the encoder and the decoder are taken to be inverses of each other in terms of network architecture. For example with image data, the encoder starts with one or more convolution layers that successively downsample followed by one or more dense layers decreasing dimensionality at each layer; the decoder starts from there and increases dimensionality at each layer starting with one or more dense layers and ending with one or more upsampling deconvolution layers to generate the image back again.

The autoencoder is not a generative model; for any x, we can find the corresponding z and then reconstruct x, but we have no way of generating new x outside of the training set. In the variational autoencoder (VAE) [1], we consider z^t as random variables sampled from a known distribution p(z) (e.g., Gaussian), and we add an extra term to the reconstruction error to enforce this. Once training is done, we can sample from this p(z) and use the decoder to generate new x.

In this paper, we extend the generative adversarial network (GAN) [2] that has recently been shown to work better than the VAE as a generative model. The original GAN model is composed of two networks, a generator G and a discriminator D (Fig. 2). Both G and D are deep neural networks with convolutional and dense layers as appropriate. The generator takes a latent vector z as input and generates an observation vector x, where z are low-dimensional and are sampled from an assumed probability distribution p(z) (e.g., multivariate Gaussian with independent features). Once training is done, we can generate new x by sampling new z from p(z) and passing them through G.

The samples generated by G are called fake; they are the adverse examples to the true x^t that we have in our training set. The aim of the discriminator is to tell the true and fake samples apart as well as possible, and that is how it is trained. The aim of the generator on the other hand is to generate fakes so well that the discriminator cannot tell them apart from the true samples. The two networks G and D play an adversarial game and gradually improve their abilities: As G gets to generate better fakes, D gets better at detecting them, which in turn forces G to get even better, and so on.

The following log-likelihood criterion is maximized by D and minimized by G:

$${\mathcal{L}}_{\text{GAN}}=\sum _{x^t\in \mathcal{X}}\log D\left(x^t\right)+\sum _{z^t\sim p\left(z\right)}\log (1-D\left(G\left(z^t\right)\right))$$

Here, x^t are the true samples drawn from the training set $\mathcal{X}$ and G(z^t) are the fake samples with z^t sampled from p(z).

Since its inception, the GAN model and its many variants have been successfully used in many applications, in image, video, text and music generation; see [3] for a survey.

Despite their various successful applications, it has been seen that training GANs is difficult and several empirical tips and tricks have been proposed to improve convergence, such as label smoothing, mini-batch discrimination, and feature matching [4].

Our approach in this paper involves adding an auxiliary loss term to that of GAN and optimize the resulting augmented criterion in training. This added term provides a “hint” that directs the learning process towards a better generator. We propose a general framework that defines how such hints can be included in training and show four variants. To be able to define the type of hint we have, we use the bidirectional form of the GAN. The original GAN can generate x for any z but does not have an inverse mapper for generating the corresponding z for a given x. The bidirectional GAN (BiGAN) [5], [6] also includes an encoder component, and this encoder allows us to define various loss functions to train better generators. This new encoder component (x → z), which is also implemented as a deep neural network, works just like the encoder of the AE, and the generator of the GAN (z → x) works just like the decoder of AE, and we use this correspondence in defining the auxiliary loss functions.

In addition to training a better generator, having an encoder can also be useful in different scenarios: Once we have such a mechanism, by investigating how z changes as x is changed, we can assign meaning to the different dimensions of z [7] and this allows knowledge extraction: For example, we can do “vector algebra” where abstract features such as putting on glasses corresponds to adding a vector in the z-space. Because such an encoder works as a dimensionality reducer trained with unlabeled data, it can be used as a preprocessor before a later classifier or regressor in a semi-supervised setting.

In Section 2.1, we discuss the BiGAN model trained using the original log-likelihood criterion as well as with the Wasserstein loss introducing the Wasserstein BiGAN. We introduce the auxiliary reconstruction criteria for training BiGANs in Section 3. Our experimental results on five image data sets are given in Section 4 and we conclude in Section 5.