Does literacy improve brain function? Does it also entail losses? Using functional magnetic resonance imaging, we measured brain responses to spoken and written language, visual faces, houses, tools, and checkers in adults of variable literacy (10 were illiterate, 22 became literate as adults, and 31 were literate in childhood). As literacy enhanced the left fusiform activation evoked by writing, it induced a small competition with faces at this location, but also broadly enhanced visual responses in fusiform and occipital cortex, extending to area V1. Literacy also enhanced phonological activation to speech in the planum temporale and afforded a top-down activation of orthography from spoken inputs. Most changes occurred even when literacy was acquired in adulthood, emphasizing that both childhood and adult education can profoundly refine cortical organization.