We propose a video stabilization algorithm based on the rotation of a virtual sphere. Unlike traditional video stabilization algorithms relying on two-dimensional motion models or reconstruction of (3D) camera motions, the proposed virtual sphere model stabilizes video by projecting each frame onto the sphere and performing corrective rotations. Specifically, matching feature points between two adjacent frames are first projected onto two virtual spheres to obtain pairs of spherical points. Then, the rotation matrix between the previous and current frame is calculated. The resulting 3D rotation matrix sequence is used to represent the camera motion, and it is smoothed using the geodesic distance on a Riemannian manifold. Finally, the difference between the smoothed and original path allows obtaining the rotation matrix that causes camera jitter, and the virtual spheres are rotated reversely to suppress jitter. Experimental results show that the proposed algorithm can effectively reduce random jitter, outperforming state-of-the-art methods.