Recent advances in generative adversarial networks (GANs) have achieved great success in automated image composition that generates new images by embedding interested foreground objects into background images automatically. On the other hand, most existing works deal with foreground objects in two-dimensional (2D) images though foreground objects in three-dimensional (3D) models are more flexible with 360-degree view freedom. This paper presents an innovative View Alignment GAN (VA-GAN) that composes new images by embedding 3D models into 2D background images realistically and automatically. VA-GAN consists of a texture generator and a differential discriminator that are inter-connected and end-to-end trainable. The differential discriminator guides to learn geometric transformation from background images so that the composed 3D models can be aligned with the background images with realistic poses and views. The texture generator adopts a novel view encoding mechanism for generating accurate object textures for the 3D models under the estimated views. Extensive experiments over two synthesis tasks (car synthesis with KITTI and pedestrian synthesis with Cityscapes) show that VA-GAN achieves high-fidelity composition qualitatively and quantitatively as compared with state-of-the-art generation methods.

中文翻译：

---

通过视图对齐实现逼真的 3D 嵌入

生成对抗网络 (GAN) 的最新进展在自动图像合成方面取得了巨大成功，该合成通过将感兴趣的前景对象自动嵌入到背景图像中来生成新图像。另一方面，大多数现有作品处理二维（2D）图像中的前景对象，尽管三维（3D）模型中的前景对象更灵活，具有 360 度视图自由度。本文提出了一种创新的视图对齐 GAN (VA-GAN)，它通过将 3D 模型真实地自动嵌入到 2D 背景图像中来合成新图像。VA-GAN 由一个纹理生成器和一个差分鉴别器组成，它们是相互连接的，并且是端到端可训练的。差分鉴别器指导从背景图像中学习几何变换，以便组合的 3D 模型可以与具有真实姿势和视图的背景图像对齐。纹理生成器采用新颖的视图编码机制，为估计视图下的 3D 模型生成准确的对象纹理。对两个合成任务（KITTI 的汽车合成和 Cityscapes 的行人合成）的大量实验表明，与最先进的生成方法相比，VA-GAN 在定性和定量上都实现了高保真合成。