Speech perception abilities vary substantially across listeners, particularly in adverse conditions including those stemming from environmental degradation (e.g., noise) or from talker-related challenges (e.g., nonnative or disordered speech). This study examined adult listeners' recognition of words in phrases produced by six talkers representing three speech varieties: a nonnative accent (Spanish-accented English), a regional dialect (Irish English), and a disordered variety (ataxic dysarthria). Semantically anomalous phrases from these talkers were presented in a transcription task and intelligibility scores, percent words correct, were compared across the three speech varieties. Three cognitive-linguistic areas-receptive vocabulary, cognitive flexibility, and inhibitory control of attention-were assessed as possible predictors of individual word recognition performance. Intelligibility scores for the Spanish accent were significantly correlated with scores for the Irish English and ataxic dysarthria. Scores for the Irish English and dysarthric speech, in contrast, were not correlated. Furthermore, receptive vocabulary was the only cognitive-linguistic assessment that significantly predicted intelligibility scores. These results suggest that, rather than a global skill of perceiving speech that deviates from native dialect norms, listeners may possess specific abilities to overcome particular types of acoustic-phonetic deviation. Furthermore, vocabulary size offers performance benefits for intelligibility of speech that deviates from one's typical dialect norms.