Detecting interaural time difference (ITD) is crucial for sound localization. The temporal accuracy required to detect ITD, and how ITD is initially encoded, continue to puzzle scientists. A fundamental question is whether the monaural inputs to the binaural ITD detectors differ only in their timing, when temporal and spectral tunings are largely inseparable in the auditory pathway. Here, we investigate the spectrotemporal selectivity of the monaural inputs to ITD detector neurons of the owl. We found that these inputs are selective for instantaneous frequency glides. Modeling shows that ITD tuning depends strongly on whether the monaural inputs are spectrotemporally matched, an effect that may generalize to mammals. We compare the spectrotemporal selectivity of monaural inputs of ITD detector neurons in vivo, demonstrating that their selectivity matches. Finally, we show that this refinement can develop through spike timing-dependent plasticity. Our findings raise the unexplored issue of time-dependent frequency tuning in auditory coincidence detectors and offer a unifying perspective.