Video frame interpolation aims to improve users’ watching experiences by generating high-frame-rate videos from low-frame-rate ones. Existing approaches typically focus on synthesizing intermediate frames using high-quality reference images. However, the captured reference frames may suffer from inevitable spatial degradations such as motion blur, sensor noise, etc. Few studies have approached the joint video enhancement problem, namely synthesizing high-frame-rate and high-quality results from low-frame-rate degraded inputs. In this paper, we propose a unified optimization framework for video frame interpolation with spatial degradations. Specifically, we develop a frame interpolation module with a pyramid structure to cyclically synthesize high-quality intermediate frames. The pyramid module features adjustable spatial receptive field and temporal scope, thus contributing to controllable computational complexity and restoration ability. Besides, we propose an inter-pyramid recurrent module to connect sequential models to exploit the temporal relationship. The pyramid module integrates the recurrent module, thus can iteratively synthesize temporally smooth results. And the pyramid modules share weights across iterations, thus it does not expand the model’s parameter size. Our model can be generalized to several applications such as up-converting the frame rate of videos with motion blur, reducing compression artifacts, and jointly super-resolving low-resolution videos. Extensive experimental results demonstrate that our method performs favorably against state-of-the-art methods on various video frame interpolation and enhancement tasks.

中文翻译：

---

通过金字塔递归框架进行视频帧内插和增强

视频帧插值旨在通过从低帧率视频中生成高帧率视频来改善用户的观看体验。现有方法通常集中于使用高质量参考图像来合成中间帧。但是，捕获的参考帧可能会遭受不可避免的空间退化，例如运动模糊，传感器噪声等。很少有研究解决联合视频增强问题，即从低帧率合成高帧率和高质量结果降级的输入。在本文中，我们提出了一个具有空间退化的视频帧插值统一优化框架。具体来说，我们开发了具有金字塔结构的帧插值模块，以循环合成高质量的中间帧。金字塔模块具有可调整的空间接收场和时间范围的特征，从而有助于可控的计算复杂性和恢复能力。此外，我们提出了一个金字塔间递归模块，以连接顺序模型以利用时间关系。金字塔模块集成了递归模块，因此可以迭代地合成时间上平滑的结果。并且金字塔模块在迭代之间共享权重，因此它不会扩展模型的参数大小。我们的模型可以推广到多种应用，例如通过运动模糊将视频的帧速率上转换，减少压缩伪像以及联合超分辨低分辨率视频。