Due to the delay in the row-wise exposure and the lack of stable support when a photographer holds a CMOS camera, video jitter and rolling shutter distortion are closely coupled degradations in the captured videos. However, previous methods have rarely considered both phenomena and usually treat them separately, with stabilization approaches that are unable to handle the rolling shutter effect and rolling shutter removal algorithms that are incapable of addressing motion shake. To tackle this problem, we propose a novel method that simultaneously stabilizes and rectifies a rolling shutter shaky video. The key issue is to estimate both inter-frame motion and intra-frame motion. Specifically, for each pair of adjacent frames, we first estimate a set of spatially variant inter-frame motions using a neighbor-motion-aware local motion model, where the classical mesh-based model is improved by introducing a new constraint to enhance the neighbor motion consistency. Then, different from other 2D rolling shutter removal methods that assume the pixels in the same row have a single intra-frame motion, we build a novel mesh-based intra-frame motion calculation model to cope with the depth variation in a mesh row and obtain more faithful estimation results. Finally, temporal and spatial motion constraints and an adaptive weight assignment strategy are considered together to generate the optimal warping transformations for different motion situations. Experimental results demonstrate the effectiveness and superiority of the proposed method when compared with other state-of-the-art methods.

中文翻译：

---

同时进行视频稳定和滚动快门移除


由于行向曝光的延迟以及摄影师手持 CMOS 相机时缺乏稳定的支撑，视频抖动和卷帘快门失真是所拍摄视频的劣化密切相关。然而，以前的方法很少考虑这两种现象，并且通常单独处理它们，稳定方法无法处理卷帘快门效果，而卷帘快门去除算法则无法解决运动抖动问题。为了解决这个问题，我们提出了一种新方法，可以同时稳定和纠正滚动快门抖动视频。关键问题是估计帧间运动和帧内运动。具体来说，对于每对相邻帧，我们首先使用邻近运动感知的局部运动模型估计一组空间变化的帧间运动，其中通过引入新的约束来增强邻近的经典基于网格的模型得到改进运动一致性。然后，与假设同一行中的像素具有单个帧内运动的其他2D滚动快门去除方法不同，我们构建了一种新颖的基于网格的帧内运动计算模型来应对网格行中的深度变化，并且获得更忠实的估计结果。最后，同时考虑时间和空间运动约束以及自适应权重分配策略，以生成针对不同运动情况的最佳扭曲变换。实验结果证明了该方法与其他最先进方法相比的有效性和优越性。