Cognitive memory involves long-term memories for facts (semantic memory) and personal experiences (episodic memory) that can be brought to mind. There is consensus that the hippocampus and related medial temporal lobe (MTL) structures are crucial for adult cognitive memory, but much less is known about their contribution to memory during infancy and childhood. We argue that the MTL is involved in memory from early in life, supporting recognition memory within the first postnatal months and recall memory within the first year. We propose that normal development involves a sequence in which a form of semantic-like memory emerges first, whereas the characteristics of episodic memory develop only later with progressive development of the hippocampus. Early bilateral injury to the hippocampus disrupts this normal pattern such that memory skills cannot develop beyond the stage of semantic memories. This review is part of the INMED/TINS special issue "Nature and nurture in brain development and neurological disorders", based on presentations at the annual INMED/TINS symposium (http://inmednet.com/).