Severe soil erosion occurs on the Loess Plateau in China, which makes the Yellow River the most sediment-laden river in the world. Construction of about 60,000 sediment check dams has remarkably controlled soil erosion on the Loess Plateau and reduced the sediment load of the middle and lower Yellow River. Nonetheless, little is known about the mechanism of erosion control and vegetation development of sediment check dams. The function of a single check dam mainly is trapping sediment, while the function of a train of check dams comprising dozens of or over hundreds of check dams in a gully encompasses controlling bed incision and reducing erosion energy. A formula was proposed to calculate the potential energy of bank failure and slope failure in a gully, which essentially constitutes the erosion energy. The erosion energy increases when gully incision occurs, which is induced by the incision of the Yellow River and its tributaries on the Loess Plateau. Sediment deposition in many gullies due to construction of check dams reduces the erosion energy to almost zero, which in turn greatly reduces soil erosion and sediment yield. Construction of check dams promotes vegetation development. The vegetation-erosion dynamics model was used to study the effect of check dams on vegetation development. Simulation results show that reforestation without check dam construction might result in an increase of vegetation cover in the first ten years and then a drop of vegetation cover to less than 10% in the later years. The check dams provide a foundation for vegetation development.