Visual understanding on construction sites by deep learning, such as semantic segmentation, is hardly mentioned in the literature due to the severe lack of labeled data sets. To resolve this issue, we collect and label 859 images, including 12 classes of objects in construction activities, from different construction sites. We then adopt DeepLabV3+ on this data set with modifications. We leverage the Cityscape data set to pretrain the model, and then fine-tune it on our collected data set. Moreover, multiple data augmentation techniques are utilized to expand the training data set. Our model reaches 0.6467 mean intersection over union (mIoU) and 92.62% mean pixel accuracy (mPA) in the out-of-sample test with the capability of processing over 45 frames per second with a resolution of pixels. In addition, we develop a synthetic robotic system integrated with red–green–blue (RGB)-depth camera for visual understanding on sites. It can detect the depth information of objects and has high potential in automated construction and visual surveillance.

中文翻译：

---

用于建筑工地视觉理解的深度语义分割

由于严重缺乏标记数据集，文献中几乎没有提到通过深度学习对建筑工地进行视觉理解，例如语义分割。为了解决这个问题，我们从不同的施工现场收集并标记了 859 张图像，包括施工活动中的 12 类对象。然后我们对该数据集进行修改后采用 DeepLabV3+。我们利用 Cityscape 数据集对模型进行预训练，然后在我们收集的数据集上对其进行微调。此外，利用多种数据增强技术来扩展训练数据集。我们的模型在样本外测试中达到 0.6467 平均交集（mIoU）和 92.62% 平均像素精度（mPA），每秒处理超过 45 帧，分辨率为像素。此外，我们开发了一种与红-绿-蓝 (RGB) 深度相机集成的合成机器人系统，用于现场视觉理解。它可以检测物体的深度信息，在自动化施工和视觉监控方面具有很高的潜力。