All vocal species, including humans, often encounter simultaneous (concurrent) vocal signals from conspecifics. To segregate concurrent signals, the auditory system must extract information regarding the individual signals from their summed waveforms. During the breeding season, nesting male midshipman fish (Porichthys notatus) congregate in localized regions of the intertidal zone and produce long-duration (>1 min), multi-harmonic signals ("hums") during courtship of females. The hums of neighboring males often overlap, resulting in acoustic beats with amplitude and phase modulations at the difference frequencies (dFs) between their fundamental frequencies (F0s) and harmonic components. Behavioral studies also show that midshipman can localize a single hum-like tone when presented with a choice between two concurrent tones that originate from separate speakers. A previous study of the neural mechanisms underlying the segregation of concurrent signals demonstrated that midbrain neurons temporally encode a beat's dF through spike synchronization; however, spectral information about at least one of the beat's components is also required for signal segregation. Here we examine the encoding of spectral differences in beat signals by midbrain neurons. The results show that, although the spike rate responses of many neurons are sensitive to the spectral composition of a beat, virtually all midbrain units can encode information about differences in the spectral composition of beat stimuli via their interspike intervals (ISIs) with an equal distribution of ISI spectral sensitivity across the behaviorally relevant dFs. Together, temporal encoding in the midbrain of dF information through spike synchronization and of spectral information through ISI could permit the segregation of concurrent vocal signals.