Temporal attributes of pitch processing at cortical and subcortical levels are differentially weighted and well-coordinated. The question is whether language experience induces functional modulation of hemispheric preference complemented by brainstem ear symmetry for pitch processing. Brainstem frequency-following and cortical pitch responses were recorded concurrently from Mandarin and English participants. A Mandarin syllable with a rising pitch contour was presented to both ears with monaural stimulation. At the cortical level, left ear stimulation in the Chinese group revealed an experience-dependent response for pitch processing in the right hemisphere, consistent with a functional account. The English group revealed a contralateral hemisphere preference consistent with a structural account. At the brainstem level, Chinese participants showed a functional leftward ear asymmetry, whereas English were consistent with a structural account. Overall, language experience modulates both cortical hemispheric preference and brainstem ear asymmetry in a complementary manner to optimize processing of temporal attributes of pitch.