This paper studies the cooperative training of two generative models for image modeling and synthesis. Both models are parametrized by convolutional neural networks (ConvNets). The first model is a deep energy-based model, whose energy function is defined by a bottom-up ConvNet, which maps the observed image to the energy. We call it the descriptor network. The second model is a generator network, which is a non-linear version of factor analysis. It is defined by a top-down ConvNet, which maps the latent factors to the observed image. The maximum likelihood learning algorithms of both models involve MCMC sampling such as Langevin dynamics. We observe that the two learning algorithms can be seamlessly interwoven into a cooperative learning algorithm that can train both models simultaneously. Specifically, within each iteration of the cooperative learning algorithm, the generator model generates initial synthesized examples to initialize a finite-step MCMC that samples and trains the energy-based descriptor model. After that, the generator model learns from how the MCMC changes its synthesized examples. That is, the descriptor model teaches the generator model by MCMC, so that the generator model accumulates the MCMC transitions and reproduces them by direct ancestral sampling. We call this scheme MCMC teaching. We show that the cooperative algorithm can learn highly realistic generative models.

中文翻译：

---

描述符和生成器网络的协作训练


本文研究了两种用于图像建模和合成的生成模型的协作训练。两种模型均通过卷积神经网络 (ConvNets) 进行参数化。第一个模型是基于深度能量的模型，其能量函数由自下而上的ConvNet定义，它将观察到的图像映射到能量。我们称之为描述符网络。第二个模型是生成器网络，它是因子分析的非线性版本。它由自上而下的 ConvNet 定义，它将潜在因素映射到观察到的图像。两种模型的最大似然学习算法都涉及 MCMC 采样，例如 Langevin 动力学。我们观察到这两种学习算法可以无缝地交织成一个可以同时训练两个模型的协作学习算法。具体来说，在协作学习算法的每次迭代中，生成器模型生成初始合成示例以初始化有限步 MCMC，该有限步 MCMC 采样并训练基于能量的描述符模型。之后，生成器模型从 MCMC 如何更改其合成示例中学习。也就是说，描述符模型通过 MCMC 来教导生成器模型，以便生成器模型累积 MCMC 转换并通过直接祖先采样来再现它们。我们称这个方案为MCMC教学。我们证明了协作算法可以学习高度真实的生成模型。