Controllable Image Generation with Semi-supervised Deep Learning and Deformable-Mean-Template Based Geometry-Appearance Disentanglement

Highlights

• Generative controllable neural-net model by explicitly disentangling geometry and appearance.

• Learn geometry variability using population-mean template and per-individual deformations.

• Learn appearance variability in image space designed to factor out geometric variability.

• Semi-supervised variational learning with limited manually-annotated attributes.

• Empirical analysis on two large public datasets, comparing several existing methods.

Abstract

Typical deep-neural-network (DNN) based generative image models often (i) show limited ability to learn a disentangled latent representation, (ii) show limited controllability leading to undesirable side effects when manipulating selected attributes during image generation, and (iii) require large attribute-annotated training sets. We propose a generative DNN model for face images by explicitly disentangling geometry and appearance modeling to achieve selective controllability of the desired attributes with less side effects. To learn geometric variability, we leverage grayscale sketch representations to learn (i) a deformable mean template representing the population-mean face geometry and (ii) a generative model of deformations to model individual face-geometry variations, using dense image registration. We learn the appearance variability in a (color-image) space that we explicitly design by factoring out the geometric variability. We propose a variational formulation to enable semi-supervised learning when manually-annotated attributes are severely limited in the training set. Results on large datasets show that, compared to schemes using deformation models or variational learning, our method significantly improves face-image model fits and facial-feature controllability even with semi-supervised learning.

Introduction

Generative modeling and representation learning are important in many fields including image analysis where deep neural networks (DNNs) can learn highly nonlinear mappings to transform low-dimensional parametric distributions to complex high-dimensional distributions in the space of images. Prevalent generative models either use variational learning, e.g., variational autoencoders (VAEs), or adversarial components, e.g., generative adversarial networks (GANs). GAN training, though, may be prone to instability [1]. Advanced deep generative modeling can simulate high-quality images, but there remain limitations in controllability to selectively manipulate a specified semantic attribute of objects without unintended side effects of modifying other attributes. Towards this end, disentangled representation learning aims for a compact representation space with uncorrelated/independent components that capture distinct semantic attributes. Unsupervised learning of a disentangled representation, although attempted before [2], [3], is theoretically impossible [2] without inductive biases on the data and the learning process. We propose semi-supervised learning for controllable image generation using a disentangled representation.

We propose a novel DNN-based framework for generative face-image modeling based on variational learning that models image attributes in a latent space, and which explicitly disentangles the major factors of variability, i.e., (i) geometry, by which we mean shape, size, pose, and location of facial features, and (ii) appearance, by which we mean color and texture features. We design for controllability of geometry and appearance attributes using separate generative DNNs, and this allows us to selectively manipulate a specific facial attribute (e.g., smiling lips) without undesirably changing other facial attributes (e.g., nose size or skin tone). We propose a deformable mean-template based model that learns the mean face geometry and the geometric variability in the population around the mean. We propose to learn the template in the space of sketches, by which we mean color images transformed to grayscale (and then contrast-inverted), using dense image registration. Subsequently, to learn the variability in appearance, we propose to first deform the color images to register their geometry to that of the mean sketch, and then learn the appearance variability in the resulting representation space. This design safeguards our two generative models for geometry and appearance from inadvertently modeling major variability in, respectively, appearance and geometry. Thus, we design the learning of geometry and appearance variability in representation spaces with reduced interdependencies, to gain controllability with fewer side effects.

This paper proposes a novel DNN-based generative model for face images by explicitly disentangling geometry and appearance modeling to achieve selective controllability of the desired attributes with less side effects. To learn geometric variability, we leverage grayscale sketch representations (devoid of color) to learn (i) a deformable mean template representing the population-mean face geometry and (ii) a generative model of deformations to model individual face-geometry variations around the mean, using dense image registration. We learn the appearance variability in a (color-image) space that we explicitly design by factoring out the geometric variability. We propose a novel variational formulation to enable semi-supervised learning when manually-annotated attributes are available for a tiny fraction of the training set. Results on CelebA [4] and LFW [5] datasets show that, compared to current methods using deformation models or variational learning, our framework significantly improves face-image model fits and facial-feature controllability even with semi-supervised learning.