We investigate the problem of learning a probabilistic distribution over three-dimensional shapes given two-dimensional views of multiple objects taken from unknown viewpoints. Our approach called projective generative adversarial network ( PrGAN ) trains a deep generative model of 3D shapes whose projections (or renderings) matches the distribution of the provided 2D views. The addition of a differentiable projection module allows us to infer the underlying 3D shape distribution without access to any explicit 3D or viewpoint annotation during the learning phase. We show that our approach produces 3D shapes of comparable quality to GANs trained directly on 3D data. Experiments also show that the disentangled representation of 2D shapes into geometry and viewpoint leads to a good generative model of 2D shapes. The key advantage of our model is that it estimates 3D shape, viewpoint, and generates novel views from an input image in a completely unsupervised manner. We further investigate how the generative models can be improved if additional information such as depth, viewpoint or part segmentations is available at training time. To this end, we present new differentiable projection operators that can be used to learn better 3D generative models. Our experiments show that PrGAN can successfully leverage extra visual cues to create more diverse and accurate shapes.

中文翻译：

---

使用对抗网络从图像集合中推断 3D 形状

我们研究了在给定从未知视点获取的多个对象的二维视图的三维形状上学习概率分布的问题。我们称为投影生成对抗网络 (PrGAN) 的方法训练 3D 形状的深度生成模型，其投影（或渲染）与提供的 2D 视图的分布相匹配。添加可微分投影模块使我们能够在学习阶段无需访问任何显式 3D 或视点注释的情况下推断底层 3D 形状分布。我们表明，我们的方法产生了与直接在 3D 数据上训练的 GAN 质量相当的 3D 形状。实验还表明，将 2D 形状分解为几何图形和视点表示可以生成良好的 2D 形状生成模型。我们模型的主要优势在于它以完全无监督的方式估计 3D 形状、视点并从输入图像生成新视图。如果在训练时可以获得深度、视点或部分分割等附加信息，我们将进一步研究如何改进生成模型。为此，我们提出了新的可微投影算子，可用于学习更好的 3D 生成模型。我们的实验表明，PrGAN 可以成功地利用额外的视觉线索来创建更加多样化和准确的形状。为此，我们提出了新的可微投影算子，可用于学习更好的 3D 生成模型。我们的实验表明，PrGAN 可以成功地利用额外的视觉线索来创建更加多样化和准确的形状。为此，我们提出了新的可微投影算子，可用于学习更好的 3D 生成模型。我们的实验表明，PrGAN 可以成功地利用额外的视觉线索来创建更加多样化和准确的形状。