Objects obscured by occluders are considered lost in the images acquired by conventional camera systems, prohibiting both visualization and understanding of such hidden objects. Non-line-of-sight methods (NLOS) aim at recovering information about hidden scenes, which could help make medical imaging less invasive, improve the safety of autonomous vehicles, and potentially enable capturing unprecedented high-definition RGB-D data sets that include geometry beyond the directly visible parts. Recent NLOS methods have demonstrated scene recovery from time-resolved pulse-illuminated measurements encoding occluded objects as faint indirect reflections. Unfortunately, these systems are fundamentally limited by the quartic intensity fall-off for diffuse scenes. With laser illumination limited by eye-safety limits, recovery algorithms must tackle this challenge by incorporating scene priors. However, existing NLOS reconstruction algorithms do not facilitate learning scene priors. Even if they did, datasets that allow for such supervision do not exist, and successful encoder-decoder networks and generative adversarial networks fail for real-world NLOS data. In this work, we close this gap by learning hidden scene feature representations tailored to both reconstruction and recognition tasks such as classification or object detection, while still relying on physical models at the feature level. We overcome the lack of real training data with a generalizable architecture that can be trained in simulation. We learn the differentiable scene representation jointly with the reconstruction task using a differentiable transient renderer in the objective, and demonstrate that it generalizes to unseen classes and unseen real-world scenes , unlike existing encoder-decoder architectures and generative adversarial networks. The proposed method allows for end-to-end training for different NLOS tasks , such as image reconstruction, classification, and object detection, while being memory-efficient and running at real-time rates. We demonstrate hidden view synthesis, RGB-D reconstruction, classification, and object detection in the hidden scene in an end-to-end fashion.

中文翻译：

---

用于非视距成像和识别的学习特征嵌入

被遮挡物遮挡的物体被认为在传统相机系统获取的图像中丢失，从而禁止可视化和理解这些隐藏的物体。非视距方法 (NLOS) 旨在恢复有关隐藏场景的信息，这有助于降低医学成像的侵入性，提高自动驾驶汽车的安全性，并有可能捕获前所未有的高清 RGB-D 数据集，其中包括超出直接可见部分的几何形状。最近的 NLOS 方法已经证明了从时间分辨脉冲照明测量中恢复场景，将被遮挡的物体编码为微弱的间接反射。不幸的是，这些系统从根本上受到漫反射场景的四次强度衰减的限制。由于激光照明受到眼睛安全限制的限制，恢复算法必须通过结合场景先验来应对这一挑战。然而，现有的 NLOS 重建算法不利于学习场景先验。即使他们这样做了，也不存在允许这种监督的数据集，并且成功的编码器-解码器网络和生成对抗网络对于现实世界的 NLOS 数据都失败了。在这项工作中，我们通过学习为重建和识别任务（如分类或对象检测）量身定制的隐藏场景特征表示来缩小这一差距，同时仍然依赖于特征级别的物理模型。我们 成功的编码器-解码器网络和生成对抗网络无法处理真实世界的 NLOS 数据。在这项工作中，我们通过学习为重建和识别任务（如分类或对象检测）量身定制的隐藏场景特征表示来缩小这一差距，同时仍然依赖于特征级别的物理模型。我们 成功的编码器-解码器网络和生成对抗网络无法处理真实世界的 NLOS 数据。在这项工作中，我们通过学习为重建和识别任务（如分类或对象检测）量身定制的隐藏场景特征表示来缩小这一差距，同时仍然依赖于特征级别的物理模型。我们克服缺乏真实训练数据的问题具有可以在模拟中训练的通用架构。我们在目标中使用可微瞬态渲染器与重建任务一起学习可微场景表示，并证明它泛化到看不见的类和看不见的真实世界场景，与现有的编码器-解码器架构和生成对抗网络不同。所提出的方法允许针对不同 NLOS 任务的端到端培训，例如图像重建、分类和对象检测，同时具有内存效率和实时运行速度。我们展示隐藏视图合成、RGB-D 重建、分类和对象检测以端到端的方式在隐藏的场景中。