The recent research explosion around Neural Radiance Fields (NeRFs) shows that there is immense potential for implicitly storing scene and lighting information in neural networks, e.g., for novel view generation. However, one major limitation preventing the widespread use of NeRFs is the prohibitive computational cost of excessive network evaluations along each view ray, requiring dozens of petaFLOPS when aiming for real-time rendering on current devices. We show that the number of samples required for each view ray can be significantly reduced when local samples are placed around surfaces in the scene. To this end, we propose a depth oracle network, which predicts ray sample locations for each view ray with a single network evaluation. We show that using a classification network around logarithmically discretized and spherically warped depth values is essential to encode surface locations rather than directly estimating depth. The combination of these techniques leads to DONeRF, a dual network design with a depth oracle network as a first step and a locally sampled shading network for ray accumulation. With our design, we reduce the inference costs by up to 48x compared to NeRF. Using an off-the-shelf inference API in combination with simple compute kernels, we are the first to render raymarching-based neural representations at interactive frame rates (15 frames per second at 800x800) on a single GPU. At the same time, since we focus on the important parts of the scene around surfaces, we achieve equal or better quality compared to NeRF.

中文翻译：

---

DONeRF：使用深度Oracle网络实现神经辐射场的实时渲染

最近关于神经辐射场（NeRF）的研究爆炸表明，在神经网络中隐式存储场景和光照信息具有巨大的潜力，例如，用于生成新颖的视图。但是，阻止NeRF广泛使用的一个主要限制是沿每个视线进行过多的网络评估所带来的过高的计算成本，在瞄准当前设备上的实时渲染时，需要数十个petaFLOPS。我们显示，当将局部样本放置在场景中的表面周围时，可以大大减少每个视线所需的样本数量。为此，我们提出了一个深度预言网络，该网络可以通过单个网络评估来预测每个视线的射线样本位置。我们表明，使用围绕对数离散和球形扭曲深度值的分类网络对于编码表面位置而不是直接估计深度至关重要。这些技术的结合产生了DONeRF，这是一个双网络设计，第一步是深度oracle网络，另一个是用于光线累积的本地采样着色网络。通过我们的设计，与NeRF相比，我们最多可将推理成本降低48倍。结合使用现成的推理API和简单的计算内核，我们是第一个在单个GPU上以交互帧速率（800x800下每秒15帧）渲染基于光线行进的神经表示的方法。同时，由于我们专注于表面周围场景的重要部分，因此与NeRF相比，我们可以达到同等或更好的质量。