Efficient rendering of photo‐realistic virtual worlds is a long standing effort of computer graphics. Modern graphics techniques have succeeded in synthesizing photo‐realistic images from hand‐crafted scene representations. However, the automatic generation of shape, materials, lighting, and other aspects of scenes remains a challenging problem that, if solved, would make photo‐realistic computer graphics more widely accessible. Concurrently, progress in computer vision and machine learning have given rise to a new approach to image synthesis and editing, namely deep generative models. Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e.g., by the integration of differentiable rendering into network training. With a plethora of applications in computer graphics and vision, neural rendering is poised to become a new area in the graphics community, yet no survey of this emerging field exists. This state‐of‐the‐art report summarizes the recent trends and applications of neural rendering. We focus on approaches that combine classic computer graphics techniques with deep generative models to obtain controllable and photorealistic outputs. Starting with an overview of the underlying computer graphics and machine learning concepts, we discuss critical aspects of neural rendering approaches. Specifically, our emphasis is on the type of control, i.e., how the control is provided, which parts of the pipeline are learned, explicit vs. implicit control, generalization, and stochastic vs. deterministic synthesis. The second half of this state‐of‐the‐art report is focused on the many important use cases for the described algorithms such as novel view synthesis, semantic photo manipulation, facial and body reenactment, relighting, free‐viewpoint video, and the creation of photo‐realistic avatars for virtual and augmented reality telepresence. Finally, we conclude with a discussion of the social implications of such technology and investigate open research problems.

中文翻译：

---

神经渲染的最新技术

逼真的虚拟世界的有效渲染是计算机图形学的长期努力。现代图形技术已成功地从手工制作的场景表示中合成逼真的图像。然而，自动生成形状、材料、照明和场景的其他方面仍然是一个具有挑战性的问题，如果解决这个问题，将使逼真的计算机图形更容易获得。同时，计算机视觉和机器学习的进步催生了一种新的图像合成和编辑方法，即深度生成模型。神经渲染是一个新兴的新兴领域，它将生成式机器学习技术与计算机图形学的物理知识相结合，例如，通过将可微渲染集成到网络训练中。随着计算机图形学和视觉领域的大量应用，神经渲染有望成为图形界的一个新领域，但目前还没有关于这个新兴领域的调查。这份最先进的报告总结了神经渲染的最新趋势和应用。我们专注于将经典计算机图形技术与深度生成模型相结合的方法，以获得可控和逼真的输出。我们从基本计算机图形学和机器学习概念的概述开始，讨论神经渲染方法的关键方面。具体来说，我们的重点是控制的类型，即如何提供控制、学习管道的哪些部分、显式与隐式控制、泛化以及随机与确定性综合。这份最新报告的后半部分重点介绍了所描述算法的许多重要用例，例如新颖的视图合成、语义照片处理、面部和身体重演、重新照明、自由视点视频和创建用于虚拟和增强现实远程呈现的照片般逼真的化身。最后，我们讨论了此类技术的社会影响并调查了开放的研究问题。