Auditory neuropathy affects the normal synchronous activity in the auditory nerve, without affecting the amplification function in the inner ear. Patients with auditory neuropathy often complain that they can hear sounds, but cannot understand speech. Here we report psychophysical tests indicating that these patients' poor speech recognition is due to a severe impairment in their temporal processing abilities. We also simulate this temporal processing impairment in normally hearing listeners and produce similar speech recognition deficits. This study demonstrates the importance of neural synchrony for auditory perceptions including speech recognition in humans. The results should contribute to better diagnosis and treatment of auditory neuropathy.