Temporal processing is crucial in many aspects of our perception and action. While there is mounting evidence for the encoding mechanisms of spatial ("where") and identity ("what") information, those of temporal information ("when") remain largely unknown. Recent studies suggested that, similarly to the basic visual stimulus features such as orientation, motion direction, and numerical quantity, event durations are also represented by a population of neurons that are tuned for specific, preferred durations. This paper first reviews recent psychophysical studies on duration aftereffect. Changes in the three parameters (response gain, shift, and width of tuning curves) are then discussed that may need to be taken into account in the putative duration-channel model. Next, the potential neural basis of the duration channels is examined by overviewing recent neuroimaging and electrophysiological studies on time perception. Finally, this paper proposes a general neural basis of timing that commonly represents time-differences independent of stimulus types (e.g., a single duration v.s. multiple brief events). This extends the idea of the "when pathway" from the perception of temporal order to the general timing mechanisms for the perception of duration, temporal frequency, and synchrony.