Whereas sensory perception relies on specialized sensory pathways, it is unclear whether these pathways originate as modality-specific circuits. We demonstrated that somatosensory and visual circuits are not by default segregated but require the earliest retinal activity to do so. In the embryo, somatosensory and visual circuits are intermingled in the superior colliculus, leading to cortical multimodal responses to whisker pad stimulation. At birth, these circuits segregate, and responses switch to unimodal. Blocking stage I retinal waves prolongs the multimodal configuration into postnatal life, with the superior colliculus retaining a mixed somato-visual molecular identity and defects arising in the spatial organization of the visual system. Hence, the superior colliculus mediates the timely segregation of sensory modalities in an input-dependent manner, channeling specific sensory cues to their appropriate sensory pathway.