Video snapshot compressive imaging (SCI) system enables high-frame-rate imaging by projecting multiple frames into a 2D snapshot measurement during a single exposure, and the original video frames can be reconstructed by solving an optimization problem. However, existing methods usually cannot achieve a good balance between reconstruction time and reconstruction quality, which has become a major obstacle for practical application of video SCI. In order to cope with this issue, we propose a residual ensemble network to learn the explicit inverse mapping from the 2D snapshot measurement to the original video. Specifically, the proposed network aims to exploit the spatiotemporal correlations between video frames for improving reconstruction quality. The spatiotemporal correlations of video frames demonstrate multiple types, including intra-frame spatial correlation, inter-frame forward and backward temporal correlation. With the purpose of fully capturing these differentiated correlations, we design four sub-networks, namely, a pseudo-3D U-shape sub-network, two residual sub-networks, and a serial forward and backward recurrent sub-network, and further assemble these four sub-networks into an ensemble network through alternate residual links. This ensemble network can effectively fuse the predictions of each sub-network and maintain spatiotemporal consistency between video frames. We further design a compound loss function to guide the network learning, and the new video can be fast reconstructed by simply feeding its 2D snapshot measurement into the learned network. The experimental results demonstrate that our network can significantly improve the reconstruction quality while maintaining low computational cost.

中文翻译：

---

使用残差集成网络的视频快照压缩成像


视频快照压缩成像（SCI）系统通过在单次曝光期间将多个帧投影到二维快照测量中来实现高帧率成像，并且可以通过解决优化问题来重建原始视频帧。然而，现有方法通常无法在重建时间和重建质量之间取得良好的平衡，这已成为视频SCI实际应用的主要障碍。为了解决这个问题，我们提出了一个残差集成网络来学习从 2D 快照测量到原始视频的显式逆映射。具体来说，所提出的网络旨在利用视频帧之间的时空相关性来提高重建质量。视频帧的时空相关性表现出多种类型，包括帧内空间相关性、帧间前向和后向时间相关性。为了充分捕捉这些差异相关性，我们设计了四个子网络，即一个伪3D U形子网络、两个残差子网络和一个串行前向和后向循环子网络，并进一步组装这四个子网络通过交替的剩余链路形成一个整体网络。该集成网络可以有效地融合每个子网络的预测并保持视频帧之间的时空一致性。我们进一步设计了一个复合损失函数来指导网络学习，并且只需将其 2D 快照测量输入到学习网络中即可快速重建新视频。实验结果表明，我们的网络可以显着提高重建质量，同时保持较低的计算成本。