Face recognition in young human adults preferentially relies on the processing of horizontally-oriented visual information. We addressed whether the horizontal tuning of face perception is modulated by the extensive experience humans acquire with faces over the lifespan, or whether it reflects an invariable processing bias for this visual category. We tested 296 subjects aged from 6 to 74 years in a face matching task. Stimuli were upright and inverted faces filtered to preserve information in the horizontal or vertical orientation, or both (HV) ranges. The reliance on face-specific processing was inferred based on the face inversion effect (FIE). FIE size increased linearly until young adulthood in the horizontal but not the vertical orientation range of face information. These findings indicate that the protracted specialization of the face processing system relies on the extensive experience humans acquire at encoding the horizontal information conveyed by upright faces.