Image-based geometric modeling and novel view synthesis based on sparse large-baseline samplings are challenging but important tasks for emerging multimedia applications such as virtual reality and immersive telepresence. Existing methods fail to produce satisfactory results due to the limitation on inferring reliable depth information over such challenging reference conditions. With the popularization of commercial light field (LF) cameras, capturing LF images (LFIs) is as convenient as taking regular photos, and geometry information can be reliably inferred. This inspires us to use a sparse set of LF captures to render high-quality novel views globally. However, the fusion of LF captures from multiple angles is challenging due to the scale inconsistency caused by various capture settings. To overcome this challenge, we propose a novel scale-consistent volume rescaling algorithm that robustly aligns the disparity probability volumes (DPV) among different captures for scale-consistent global geometry fusion. Based on the fused DPV projected to the target camera frustum, novel learning-based modules (i.e., the attention-guided multi-scale residual fusion module, and the disparity field-guided deep re-regularization module), which comprehensively regularize noisy observations from heterogeneous captures for high-quality rendering of novel LFIs, have been proposed. Both quantitative and qualitative experiments over the Stanford Lytro Multi-view LF dataset show that the proposed method outperforms state-of-the-art methods significantly under different experiment settings for disparity inference and LF synthesis.

中文翻译：

---

尺度一致的融合：从异构局部采样到全局沉浸式渲染

基于图像的几何建模和基于稀疏大基线采样的新颖视图合成对于虚拟现实和沉浸式远程呈现等新兴多媒体应用来说是具有挑战性但重要的任务。由于在这种具有挑战性的参考条件下推断可靠深度信息的限制，现有方法未能产生令人满意的结果。随着商用光场 (LF) 相机的普及，捕捉 LF 图像 (LFI) 与拍摄普通照片一样方便，并且可以可靠地推断几何信息。这激发了我们使用稀疏的 LF 捕获集来在全球范围内渲染高质量的小说视图。然而，由于各种捕获设置导致的尺度不一致，从多个角度融合 LF 捕获具有挑战性。为了克服这一挑战，我们提出了一种新颖的尺度一致体积重缩放算法，该算法可以稳健地对齐不同捕获之间的视差概率体积 (DPV)，以实现尺度一致的全局几何融合。基于投影到目标相机平截头体的融合 DPV，新颖的基于学习的模块（即注意力引导的多尺度残差融合模块和视差场引导的深度重正则化模块），全面正则化来自已经提出了用于高质量渲染新型 LFI 的异构捕获。斯坦福 Lytro 多视图 LF 数据集的定量和定性实验表明，在视差推断和 LF 合成的不同实验设置下，所提出的方法明显优于最先进的方法。