The idea that verbal and non-verbal forms of memory are segregated in their entirety, and localized to the left and right hippocampi, is arguably the most influential concept in the neuropsychology of temporal lobe epilepsy, forming a cornerstone of pre-surgical decision making, and a frame for interpreting postoperative outcome. This critical review begins by examining some of the unexpressed but inescapable assumptions of the material-specificity model: (i) verbal and non-verbal memory are unitary and internally homogenous constructs; and (ii) left and right memory systems are assumed to be independent, self-contained modules. The next section traces the origins of an alternative view, emanating largely from three challenges to these assumptions: (i) verbal memory is systematically fractionated by left mesial temporal foci; (ii) the resulting components are differentially localized within the left temporal lobe; and (iii) verbal and non-verbal memory functions are not entirely lateralized. It is argued here that the perirhinal cortex is a key node in a more extensive network mediating protosemantic associative memory. Impairment of this fundamental memory system is a proximal neurocognitive marker of mesial temporal epileptogenesis.