In terms of the target with micromotion parts, the readability of inverse synthetic aperture radar (ISAR) image of the main body is influenced by micro-Doppler (m-D) effects. Some facts, such as radar working modes and electromagnetic environment, may lead to sparse aperture, which further induces high side lobes and makes the removal of m-D effects more difficult. To jointly eliminate m-D effects and the interference introduced by sparse aperture, a novel m-D effects removed ISAR imaging algorithm is proposed. In this technique, sparse ISAR imaging with the removal of m-D effects can be established as an optimization model of joint low-rank and sparse representation, which is a quadruple constraint optimization problem, including the low rank of the high-resolution range profile (HRRP) of the main body, the sparsity of the HRRP of micromotion parts, the sparsity of the imaging result of the main body, and the noise constraint of the echo signal. Furthermore, the matrix factorization method is theoretically presented to simplify the optimization process of the nuclear norm, and the alternating direction method of multipliers is utilized to solve all constructed models efficiently. Experimental results based on both simulated and measured data demonstrate the superiority of the proposed algorithm.