Grazing exclusion is proposed as an effective strategy for to recover degraded grasslands and increase carbon accumulation. However, few studies have explored the response of soil organic carbon (SOC) and its components to grazing exclusion during long-term succession. In this study, we collected soils from sites with a chronosequence of ~33 years after grazing exclusion with randomized and replicated experiments in the temperate grasslands of the Loess Plateau, China. The contents of SOC, free particulate organic carbon (fPOC), occluded particulate organic carbon (oPOC), and mineral-associated organic carbon (MOC) and their driving factors in the 0–60 cm soil layer were examined in each grassland. The results showed that fPOC, oPOC, and MOC contents were significantly affected (P < 0.05) by grazing exclusion duration and soil depth. Soil fPOC, oPOC, POC, and MOC contents increased with exclusion time, but decreased with soil depth. SOC content was significantly improved since grazing exclusion for more than 26 years compared with that for active grazing grasslands. The fPOC played critical roles in promoting the SOC formation within the entire soil profile (0–60 cm depth). Across the sampled areas, we found a positive association between the concentrations of SOC and its fractions, and the input of litter to the topsoil (0–40 cm), and soil extracellular enzymes at greater depths (40–60 cm). Our results indicate the potential of grazing exclusion for promoting SOC accumulation in semiarid regions, and highlight the need to understand soil carbon stabilization mechanisms in both top- and subsoil under future climate change scenarios.