Neural processes underlying pitch perception at the level of the cerebral cortex are influenced by language experience. We investigated whether early, pre-attentive stages of pitch processing at the level of the human brainstem may also be influenced by language experience. The human frequency following response (FFR), reflecting sustained phase-locked activity in a population of neural elements, was used to measure activity within the rostral brainstem. FFRs elicited by four Mandarin tones were recorded from native speakers of Mandarin Chinese and English. Pitch strength (reflecting robustness of neural phase-locking at the pitch periods) and accuracy of pitch tracking were extracted from the FFRs using autocorrelation algorithms. These measures revealed that the Chinese group exhibits stronger pitch representation and smoother pitch tracking than the English group. Consistent with the pitch data, FFR spectral data showed that the Chinese group exhibits stronger representation of the second harmonic relative to the English group across all four tones. These results cannot be explained by a temporal pitch encoding scheme which simply extracts the dominant interspike interval. Rather, these results support the possibility of neural plasticity at the brainstem level that is induced by language experience that may be enhancing or priming linguistically relevant features of the speech input.