In projector-camera systems, light transport models the propagation from projector emitted radiance to camera-captured irradiance. In this paper, we propose the first end-to-end trainable solution named Deep Light Transport (DeLTra) that estimates radiometrically uncalibrated projector-camera light transport. DeLTra is designed to have two modules: DepthToAtrribute and ShadingNet. DepthToAtrribute explicitly learns rays, depth and normal, and then estimates rough Phong illuminations. Afterwards, the CNN-based ShadingNet renders photorealistic camera-captured image using estimated shading attributes and rough Phong illuminations. A particular challenge addressed by DeLTra is occlusion, for which we exploit epipolar constraint and propose a novel differentiable direct light mask. Thus, it can be learned end-to-end along with the other DeLTra modules. Once trained, DeLTra can be applied simultaneously to three projector-camera tasks: image-based relighting, projector compensation and depth/normal reconstruction. In our experiments, DeLTra shows clear advantages over previous arts with promising quality and meanwhile being practically convenient.

中文翻译：

---

DeLTra：投影仪-摄像机系统的深光传输

在投影仪-相机系统中，光传输模拟了从投影仪发射的辐射到相机捕获的辐射的传播。在本文中，我们提出了第一个名为 Deep Light Transport (DeLTra) 的端到端可训练解决方案，该解决方案可以估算未校准的投影仪-相机光传输。DeLTra 被设计成有两个模块：DepthToAtrribute 和 ShadingNet。DepthToAtrribute 明确地学习光线、深度和法线，然后估计粗略的 Phong 照明。之后，基于 CNN 的 ShadingNet 使用估计的阴影属性和粗略的 Phong 照明渲染逼真的相机捕获图像。DeLTra 解决的一个特殊挑战是遮挡，为此我们利用对极约束并提出了一种新颖的可微直接光掩模。因此，它可以与其他 DeLTra 模块一起进行端到端学习。经过训练后，DeLTra 可以同时应用于三项投影仪-摄像机任务：基于图像的重新照明、投影仪补偿和深度/法线重建。在我们的实验中，DeLTra 与以前的艺术相比显示出明显的优势，具有良好的质量，同时实用方便。