Adults across cultures speak to infants in a specific infant-directed manner. We asked whether infants use this manner of speech (infant- or adult-directed) to guide their subsequent visual preferences for social partners. We found that 5-month-old infants encode an individuals' use of infant-directed speech and adult-directed speech, and use this information to guide their subsequent visual preferences for individuals even after the speech behavior has ended. Use of infant-directed speech may act as an effective cue for infants to select appropriate social partners, allowing infants to focus their attention on individuals who will provide optimal care and opportunity for learning. This selectivity may play a crucial role in establishing the foundations of social cognition.