Understanding the formation of the Qilian Shan in the NE Tibetan Plateau provides insights into the growth mechanisms of the northern portion of the plateau through time. The onset time of exhumation in the Qilian Shan is still debated. The Qinghai Nan Shan subrange, located in the southeastern Qilian Shan, is cut by the Yellow River that forms Longyang Gorge, providing a good vertical profile for thermochronological investigation of exhumation. In this paper, we reconstruct the cooling and exhumation history of the Eastern Qinghai Nan Shan using apatite fission-track(AFT) and apatite (UTh)/He(AHe) thermochronological dating on basement rocks. The fission-track data display a marked age-elevation trend from Early Cretaceous (114–122 Ma) ages at the bottom of the canyon to Early-Middle Jurassic ages (165–180 Ma) at the top. Mean (UTh)/He ages are mainly Early Cretaceous (97–128 Ma) with one Eocene outlier (42 Ma). Combined with fission track lengths, D_par values and thermal modeling results, these data indicate that the study area resided above the closure temperature of AFT and AHe until Early Cretaceous. Rapid cooling occurred during the Early Cretaceous and early Cenozoic likely driven by the Lhasa and Indian blocks collision with the southern margin of the Asian continent. Nonetheless, since then exhumation was limited to less than a few kilometers in the study area. Combined with regional geological studies, the thermochronological analysis demonstrates that, since the Mesozoic, intense exhumation occurred simultaneously along the northern and southern margins of the Qilian Shan. Hence, we suggest that the geomorphic framework of the northeast margin of the Tibetan Plateau was established during the Early Cretaceous-early Cenozoic.