Regions of human ventral extrastriate visual cortex develop specializations for natural categories (e.g., faces) and cultural artifacts (e.g., words). In adults, category-based specializations manifest as greater neural responses in visual regions of the brain (e.g., fusiform gyrus) to some categories over others. However, few studies have examined how these specializations originate in the brains of children. Moreover, it is as yet unknown whether the development of visual specializations hinges on "increases" in the response to the preferred categories, "decreases" in the responses to nonpreferred categories, or "both." This question is relevant to a long-standing debate concerning whether neural development is driven by building up or pruning back representations. To explore these questions, we measured patterns of visual activity in 4-year-old children for 4 categories (faces, letters, numbers, and shoes) using functional magnetic resonance imaging. We report 2 key findings regarding the development of visual categories in the brain: 1) the categories "faces" and "symbols" doubly dissociate in the fusiform gyrus before children can read and 2) the development of category-specific responses in young children depends on cortical responses to nonpreferred categories that decrease as preferred category knowledge is acquired.