Prior studies on development of functional specialization in human brain mainly focus on age-related increases in regional activation and connectivity among regions. However, a few recent studies on the face network demonstrate age-related decrease in face-specialized activation in the extended face network (EFN), in addition to increase in activation in the core face network (CFN). Here we used a voxel-based global brain connectivity approach to investigate whether development of the face network exhibited both increase and decrease in network connectivity. We found the voxel-wise resting-state functional connectivity (FC) within the CFN increased with age in bilateral posterior superior temporal sulcus, suggesting the integration of the CFN during development. Interestingly, the FC of the voxels in the EFN to the right fusiform face area and occipital face area decreased with age, suggesting that the CFN segregated from the EFN during development. Moreover, the age-related connectivity in the CFN was related to behavioral performance in face processing. Overall, our study demonstrated developmental reorganization of the face network achieved by both integration within the CFN and segregation of the CFN from the EFN, which may account for the simultaneous increases and decreases in neural activation during the development of the face network.