We present a novel motion-based multiframe image super-resolution (SR) algorithm using a convolutional neural network (CNN) that fuses multiple interpolated input frames to produce an SR output. We refer to the proposed CNN and associated preprocessing as the Fusion of Interpolated Frames Network (FIFNET). We believe this is the first such CNN approach in the literature to perform motion-based multiframe SR by fusing multiple input frames in a single network. We study the FIFNET using translational interframe motion with both fixed and random frame shifts. The input to the network is a sequence of interpolated and aligned frames. One key innovation is that we compute subpixel interframe registration information for each interpolated pixel and feed this into the network as additional input channels. We demonstrate that this subpixel registration information is critical to network performance. We also employ a realistic camera-specific optical transfer function model that accounts for diffraction and detector integration when generating training data. We present a number of experimental results to demonstrate the efficacy of the proposed FIFNET using both simulated and real camera data. The real data come directly from a camera and are not artificially downsampled or degraded. In the quantitative results with simulated data, we show that the FIFNET performs favorably in comparison to the benchmark methods tested.

我们提出了一种新颖的基于运动的多帧图像超分辨率（SR）算法，该算法使用了卷积神经网络（CNN），该算法融合了多个内插的输入帧以生成SR输出。我们将提出的CNN和相关的预处理称为内插帧网络融合（FIFNET）。我们认为，这是文献中第一个通过在单个网络中融合多个输入帧来执行基于运动的多帧SR的CNN方法。我们使用固定和随机帧移位的平移帧间运动研究FIFNET。网络的输入是一系列内插和对齐的帧。一个关键的创新是我们为每个插值像素计算子像素帧间配准信息，并将其作为额外的输入通道馈入网络。我们证明此子像素配准信息对于网络性能至关重要。我们还采用了特定于相机的逼真的光学传递函数模型，该模型在生成训练数据时考虑了衍射和检测器集成。我们提出了许多实验结果，以使用模拟和真实相机数据来证明所提出的FIFNET的功效。真实数据直接来自相机，不会人为地进行下采样或降级。在带有模拟数据的定量结果中，我们表明FIFNET与测试的基准方法相比表现良好。我们提出了许多实验结果，以使用模拟和真实相机数据来证明所提出的FIFNET的功效。真实数据直接来自相机，不会人为地进行下采样或降级。在带有模拟数据的定量结果中，我们表明FIFNET与测试的基准方法相比表现良好。我们提出了许多实验结果，以使用模拟和真实相机数据来证明所提出的FIFNET的功效。真实数据直接来自相机，不会人为地进行下采样或降级。在带有模拟数据的定量结果中，我们表明FIFNET与测试的基准方法相比表现良好。