In this paper, we propose a novel semantic segmentation-based stereo reconstruction method that can keep up with the accuracy of the state-of-the art approaches while running in real time. The solution follows the classic stereo pipeline, each step in the stereo workflow being enhanced by additional information from semantic segmentation. Therefore, we introduce several improvements to computation, aggregation, and optimization by adapting existing techniques to integrate additional surface information given by each semantic class. For the cost computation and optimization steps, we propose new genetic algorithms that can incrementally adjust the parameters for better solutions. Furthermore, we propose a new post-processing edge-aware filtering technique relying on an improved convolutional neural network (CNN) architecture for disparity refinement. We obtain the competitive results at 30 frames/s, including segmentation.

中文翻译：

---

基于实时语义分割的立体重建

在本文中，我们提出了一种新的基于语义分割的立体重建方法，该方法可以在实时运行时跟上最先进方法的准确性。该解决方案遵循经典的立体管道，立体工作流程中的每一步都通过语义分割的附加信息得到增强。因此，我们通过调整现有技术来集成每个语义类给出的附加表面信息，从而对计算、聚合和优化进行了一些改进。对于成本计算和优化步骤，我们提出了新的遗传算法，可以逐步调整参数以获得更好的解决方案。此外，我们提出了一种新的后处理边缘感知过滤技术，该技术依赖于改进的卷积神经网络 (CNN) 架构进行视差细化。我们以 30 帧/秒的速度获得了有竞争力的结果，包括分割。