To guarantee vessel traffic safety in inland waterways, the automatic identification system (AIS) and shore-based cameras have been widely adopted to monitor moving vessels. The AIS data could provide the unique maritime mobile service identity (MMSI), position coordinates (i.e., latitude and longitude), course over ground, and speed over ground for the vessels of interest. In contrast, the cameras could directly display the visual appearance of vessels but fail to accurately grasp the vessels’ identity information and motion parameters. In this paper, we propose to improve the maritime traffic surveillance in inland waterways using the robust fusion of AIS and visual data. It is able to obtain more accurate vessel tracking results and kinematic characteristics. In particular, to robustly track the visual vessels under complex scenarios, we first propose an anti-occlusion vessel tracking method based on the simple online and real-time tracking with a deep association metric (DeepSORT) method. We then preprocess and predict the vessel positions to obtain synchronous AIS and visual data. Before the implementation of AIS and visual data fusion, the AIS position coordinates in the geodetic coordinate system will be projected into the image coordinate system via the coordinate transformation. A multi-feature similarity measurement-based Hungarian algorithm is finally proposed to robustly and accurately fuse the AIS and visual data in the image coordinate system. For the sake of repeating fusion experiments, we have also presented a new multi-sensor dataset containing AIS data and shore-based camera imagery. The quantitative and qualitative experiments show that our fusion method is capable of improving the maritime traffic surveillance in inland waterways. It can overcome the vessel occlusion problem and fully utilizes the advantages of multi-source data to promote the maritime surveillance, resulting in enhanced vessel traffic safety and efficiency. In this work, the presented multi-sensor dataset and source code are available at https://github.com/QuJX/AIS-Visual-Fusion.

为保障内河航道的船舶通行安全，自动识别系统（AIS）和岸基摄像头已被广泛用于监控移动的船舶。AIS 数据可以为感兴趣的船只提供唯一的海上移动服务标识 (MMSI)、位置坐标（即纬度和经度）、对地航向和对地航速。相比之下，摄像头可以直接显示船只的视觉外观，但无法准确掌握船只的身份信息和运动参数。在本文中，我们建议使用 AIS 和视觉数据的稳健融合来改进内陆水道的海上交通监控。它能够获得更准确的血管跟踪结果和运动学特性。特别是，为了在复杂场景下稳健地跟踪视觉血管，我们首先提出了一种基于简单在线实时跟踪的抗闭塞血管跟踪方法，深度关联度量（DeepSORT）方法。然后我们预处理和预测船只位置以获得同步 AIS 和视觉数据。在实现AIS与视觉数据融合之前，AIS在大地坐标系中的位置坐标将通过坐标变换投影到图像坐标系中。最后提出了一种基于多特征相似性度量的匈牙利算法，以稳健、准确地融合图像坐标系中的 AIS 和视觉数据。为了重复融合实验，我们还提出了一个新的多传感器数据集，其中包含 AIS 数据和岸基摄像机图像。定量和定性实验表明，我们的融合方法能够改善内陆水道的海上交通监控。它可以克服船舶遮挡问题，充分利用多源数据的优势促进海事监督，从而提高船舶通行安全和效率。在这项工作中，提供的多传感器数据集和源代码可在https://github.com/QuJX/AIS-Visual-Fusion。