Volume Rendering is an important technique for visualizing three-dimensional scalar data grids and is commonly employed for scientific and medical image data. Direct Volume Rendering (DVR) is a well established and efficient rendering algorithm for volumetric data. Neural rendering uses deep neural networks to solve inverse rendering tasks and applies techniques similar to DVR. However, it has not been demonstrated successfully for the rendering of scientific volume data. In this work, we introduce Deep Direct Volume Rendering (DeepDVR), a generalization of DVR that allows for the integration of deep neural networks into the DVR algorithm. We conceptualize the rendering in a latent color space, thus enabling the use of deep architectures to learn implicit mappings for feature extraction and classification, replacing explicit feature design and hand-crafted transfer functions. Our generalization serves to derive novel volume rendering architectures that can be trained end-to-end directly from examples in image space, obviating the need to manually define and fine-tune multidimensional transfer functions while providing superior classification strength. We further introduce a novel stepsize annealing scheme to accelerate the training of DeepDVR models and validate its effectiveness in a set of experiments. We validate our architectures on two example use cases: (1) learning an optimized rendering from manually adjusted reference images for a single volume and (2) learning advanced visualization concepts like shading and semantic colorization that generalize to unseen volume data. We find that deep volume rendering architectures with explicit modeling of the DVR pipeline effectively enable end-to-end learning of scientific volume rendering tasks from target images.

中文翻译：

---

深度直接体积渲染：从示例图像中学习视觉特征映射

体绘制是一种用于可视化三维标量数据网格的重要技术，通常用于科学和医学图像数据。直接体积渲染 (DVR) 是一种成熟且高效的体积数据渲染算法。神经渲染使用深度神经网络来解决逆渲染任务，并应用类似于 DVR 的技术。但是，它尚未成功用于渲染科学体数据。在这项工作中，我们引入了深度直接体积渲染 (DeepDVR)，这是 DVR 的一种推广，允许将深度神经网络集成到 DVR 算法中。我们将潜在色彩空间中的渲染概念化，从而能够使用深层架构来学习用于特征提取和分类的隐式映射，取代显式特征设计和手工制作的传递函数。我们的泛化用于推导出新颖的体绘制架构，可以直接从图像空间中的示例进行端到端的训练，无需手动定义和微调多维传递函数，同时提供卓越的分类强度。我们进一步引入了一种新颖的步长退火方案，以加速 DeepDVR 模型的训练并在一组实验中验证其有效性。我们在两个示例用例中验证了我们的架构：(1) 从手动调整的单个体积的参考图像中学习优化渲染，以及 (2) 学习高级可视化概念，如泛化到看不见的体积数据的着色和语义着色。