The sensitivity of size perception to context has been used to distinguish between 'vision for action' and 'vision for perception', and to study cultural, psychopathological, and developmental differences in perception. The status of that evidence is much debated, however. Here we use a rigorous double dissociation paradigm based on the Ebbinghaus illusion, and find that for children below 7 years of age size discrimination is much less affected by surround size. Young children are less accurate than adults when context is helpful, but more accurate when context is misleading. Even by the age of 10 years context-sensitivity is still not at adult levels. Therefore, size contrast as shown by the Ebbinghaus illusion is not a built-in property of the ventral pathway subserving vision for perception but a late development of it, and low sensitivity to the Ebbinghaus illusion in autism is not primary to the pathology. Our findings also show that, although adults in Western cultures have low context-sensitivity relative to East Asians, they have high context-sensitivity relative to children. Overall, these findings reveal a gradual developmental trend toward ever broader contextual syntheses. Such developments are advantageous, but the price paid for them is that, when context is misleading, adults literally see the world less accurately than they did as children.