Auditory nerve responses to electrical stimulation exhibit aberrantly synchronous response latencies to low-rate pulse trains, nevertheless, cochlear implant users generally have elevated inter-aural timing difference detection thresholds. These findings present an apparent paradox in which single units are unusually precise but downstream within the auditory pathway access to this precision is lost. Auditory nerves innervating a region of cochlea exhibit natural heterogeneity in their diameter, myelination, and other structural properties; a key question is whether this diversity may contribute to the loss of temporal fidelity. In this work, responses of simulated auditory neuron populations with realistic intrinsic diameter and myelination heterogeneity to low-rate pulse trains were produced. By performing a receiver operating characteristic analysis on response latency distributions, ideal-observer interaural timing difference (ITD) detection limits were produced for each population. Fiber heterogeneity produced dispersion of inter-fiber latencies that produced ITD thresholds like that observed in the best performing cochlear implant users. Incorporation of myelin loss into these populations further increased inter-fiber latency variance and elevated ITD detection limits. These findings suggest that the interaction of applied currents with fibers' specific intrinsic properties may introduce fundamental limits on presentation of fine temporal structure in electrical stimulation.