Natural images inevitably suffer from spatially variant blur caused by the relative motion between a camera and objects. We present an effective and efficient patch-wise edge-enhanced image regularization and a robust kernel similarity constraint to perform an accurate kernel estimation from coarse-to-fine iterations. The proposed adaptive regularization introduces a gradient magnitude penalty function into total variation to preserve and enhance salient edges while smoothing out harmful subtle structures. In addition, the similarity constraint is engaged in each patch without camera rotation effects, ensuring that the erroneous kernels can be identified by measuring the similarity among the kernels of neighbor patches and be replaced with the well-estimated ones. After obtaining accurate kernels, numerous nonblind deblurring methods can be applied to restore an image. Numerical experiments demonstrate that the proposed algorithm performs favorably without ringing artifacts and possesses high processing efficiency for natural nonuniform blurred images.