With recent advances in sensor technology, multispectral systems are becoming increasingly attractive for intelligence, surveillance, and reconnaissance applications. Fusing information from multiple imaging modalities is a major task for such systems. Combining feature maps obtained from multiple deep neural network pipelines demonstrates promising performance for object detection and tracking. However, feature fusion using multiple deep networks is computationally intensive and therefore not suitable for resource-constrained IoT edge devices. In this paper, we propose a novel method to fuse the input space to enable processing of multispectral data via a single deep network. We use task-driven feedback as a reward signal for our reinforcement learning-based multispectral input fusion. Proposed approach not only improves tracking accuracy but also maximizes modality-specific information as intended by the user.

中文翻译：

---

多光谱信息融合与强化学习用于物联网边缘设备中的对象跟踪

随着传感器技术的最新进展，多光谱系统对情报、监视和侦察应用的吸引力越来越大。融合来自多种成像模式的信息是此类系统的一项主要任务。组合从多个深度神经网络管道获得的特征图展示了对象检测和跟踪的良好性能。然而，使用多个深度网络的特征融合是计算密集型的，因此不适合资源受限的物联网边缘设备。在本文中，我们提出了一种融合输入空间的新方法，以通过单个深度网络处理多光谱数据。我们使用任务驱动的反馈作为我们基于强化学习的多光谱输入融合的奖励信号。