As a potential solution to next-generation nanolithography, directed self-assembly (DSA) of block copolymers (BCPs) is still restrained in high-volume manufacturing primarily due to its defectivity issue. Though defects possess greater free energies than aligned morphologies and are highly energetically unfavorable, they can be kinetically trapped by the energy barriers and persist for a long time during annealing. Therefore, understanding the kinetics of defect annihilation is crucial in revealing the mechanism of defect formation and in further reducing defectivity in DSA. We focus on two types of predominant defects in DSA—dislocation and bridge. A kinetic model of each defect type is developed through statistical analysis of experimental data, providing insight into possible approaches of further defect reduction. We also investigate the impact of annealing temperature and film thickness on annihilation kinetics and discuss the reasons behind the observed results. By simply optimizing annealing conditions and film thickness, we have successfully reduced the total defect density by 1 order of magnitude. Though these findings are based on polystyrene-b-poly(methyl methacrylate) (PS-b-PMMA), we anticipate they could be readily applied to other BCP platforms as well.