Parallel processing streams in the primate visual system originate from more than a dozen anatomically and functionally distinct types of retinal ganglion cells (RGCs). A central problem in determining how visual information is processed is understanding how each of these RGC types connects to more central structures, including the lateral geniculate nucleus (LGN) of the thalamus and (via the LGN) the primary visual cortex. Nevertheless, the available functional and anatomical evidence linking together specific cell types across these structures is surprisingly indirect. This review evaluates the available evidence and assesses the strength of the many inferences that can be made from these observations. There is strong evidence that parasol RGCs are the provenance of the magnocellular (M) visual pathway and that midget RGCs give rise to the parvocellular (P) pathway. Furthermore, the M and P pathways remain segregated up to the input layer of primary visual cortex. The relationships between the numerous other RGC types and cell types in the LGN remain less certain. and there remains ambiguity about how best to define additional pathways, such as the koniocellular (K) pathway, which probably arise from these other, less common, RGC types.