Multispectral images are increasingly used for pedestrian detection. Preliminary fusion strategies would fail to exploit informative features from cross-spectral images, or worse, may introduce additional interference. In this paper, we propose an attention based multi-layer fusion network in the triple-stream deep convolutional neural network architecture for multispectral pedestrian detection. The effectiveness of multi-layer fusion is examined and verified in this work. Furthermore, a channel-wise attention module (CAM) and a spatial-wise attention module (SAM) are developed and incorporated into the network aiming at more subtle adjustment to weights of multispectral features along both the channel and spatial dimensions respectively. Channel-wise attention is trained with self-supervision while spatial-wise attention is trained with external supervision as we remodel its learning process as saliency detection. Both attention-based weighting mechanisms are evaluated separately and then sequentially. Experimental results on the KAIST dataset show that the proposed multi-layer cross-spectral fusion R-CNN (CS-RCNN), with spatial-wise weighting applied alone, achieves state-of-the-art performance on all-day detection while outperforming compared methods at nighttime.

中文翻译：

---

基于注意力的多光谱图像多层融合用于行人检测

多光谱图像越来越多地用于行人检测。初步融合策略将无法利用来自交叉光谱图像的信息特征，或者更糟的是，可能会引入额外的干扰。在本文中，我们在三流深度卷积神经网络架构中提出了一种基于注意力的多层融合网络，用于多光谱行人检测。在这项工作中检查和验证了多层融合的有效性。此外，开发了通道注意模块（CAM）和空间注意模块（SAM）并将其合并到网络中，旨在分别沿通道和空间维度对多光谱特征的权重进行更细微的调整。当我们将其学习过程重构为显着性检测时，通道注意力是通过自我监督训练的，而空间注意力是通过外部监督来训练的。两种基于注意力的加权机制都分别进行评估，然后依次进行评估。在 KAIST 数据集上的实验结果表明，所提出的多层交叉光谱融合 R-CNN (CS-RCNN)，单独应用空间加权，在全天检测中实现了最先进的性能，同时表现出色比较夜间的方法。