We present a practical and inexpensive method to reconstruct 3D scenes that include transparent and mirror objects. Our work is motivated by the need for automatically generating 3D models of interior scenes, which commonly include glass. These large structures are often invisible to cameras or even to our human visual system. Existing 3D reconstruction methods for transparent objects are usually not applicable in such a room-sized reconstruction setting. Our simple hardware setup augments a regular depth camera (e.g., the Microsoft Kinect camera) with a single ultrasonic sensor, which is able to measure the distance to any object, including transparent surfaces. The key technical challenge is the sparse sampling rate from the acoustic sensor, which only takes one point measurement per frame. To address this challenge, we take advantage of the fact that the large scale glass structures in indoor environments are usually either piece-wise planar or a simple parametric surface. Based on these assumptions, we have developed a novel sensor fusion algorithm that first segments the (hybrid) depth map into different categories such as opaque/transparent/infinity (e.g., too far to measure) and then updates the depth map based on the segmentation outcome. We validated our algorithms with a number of challenging cases, including multiple panes of glass, mirrors, and even a curved glass cabinet.

中文翻译：

---

通过声学和视觉融合在玻璃和镜子存在下进行 3D 重建


我们提出了一种实用且廉价的方法来重建包含透明和镜像对象的 3D 场景。我们的工作动机是自动生成室内场景 3D 模型（通常包括玻璃）的需求。这些大型结构通常对于相机甚至我们人类的视觉系统来说是不可见的。现有的透明物体 3D 重建方法通常不适用于这种房间大小的重建环境。我们简单的硬件设置通过单个超声波传感器增强了常规深度相机（例如 Microsoft Kinect 相机），该传感器能够测量到任何物体（包括透明表面）的距离。关键的技术挑战是声学传感器的稀疏采样率，每帧仅进行一个点测量。为了应对这一挑战，我们利用室内环境中的大型玻璃结构通常是分段平面或简单参数化表面的事实。基于这些假设，我们开发了一种新颖的传感器融合算法，该算法首先将（混合）深度图分割成不同的类别，例如不透明/透明/无穷大（例如，太远而无法测量），然后根据分割更新深度图结果。我们用许多具有挑战性的案例验证了我们的算法，包括多层玻璃、镜子，甚至弧形玻璃柜。