In real crystals and at finite temperatures point defects are inevitable. Under shear their dynamics severely influence the mechanical properties of these crystals, giving rise to non-linear effects, such as ductility. In an effort to elucidate the complex behavior of crystals under plastic deformation it is crucial to explore and to understand the interplay between the timescale related to the equilibrium point-defect diffusion and the shear-induced timescale. Based on extensive non-equilibrium molecular dynamics simulations we present a detailed investigation on the yielding behavior of cluster crystals, an archetypical model for a defect-rich crystal: in such a system clusters of overlapping particles occupy the lattice sites of a regular (FCC) structure. In equilibrium particles diffuse via site-to-site hopping while maintaining the crystalline structure intact. We investigate these cluster crystals at a fixed density and at different temperatures where the system remains in the FCC structure: temperature allows us to vary the diffusion timescale appropriately. We then expose the crystal to shear, thereby choosing shear rates which cover timescales that are both higher and lower than the equilibrium diffusion timescales. We investigate the macroscopic and microscopic response of our cluster crystal to shear and find that the yielding scenario of such a system does not rely on the diffusion of the particles – it is rather related to the plastic deformation of the underlying crystalline structure. The local bond order parameters and the measurement of local angles between neighboring clusters confirm the cooperative movement of the clusters close to the yield point. Performing complementary, related simulations for an FCC crystal formed by harshly repulsive particles reveals similarities in the yielding behavior between both systems. Still we find that the diffusion of particles does influence characteristic features in the cluster crystal, such as a less prominent increase of order parameters close to the yield point. Our simulations provide for the first time an insight into the role of the diffusion of defects in the yielding behavior of a defect-rich crystal under shear. These observations will thus be helpful in the development of theories for the plastic deformation of defect-rich crystals.