While sarcasm can be conveyed solely through contextual cues such as counterfactual or echoic statements, face-to-face sarcastic speech may be characterized by specific paralinguistic features that alert the listener to interpret the utterance as ironic or critical, even in the absence of contextual information. We investigated the neuroanatomy underlying failure to understand sarcasm from dynamic vocal and facial paralinguistic cues. Ninety subjects (20 frontotemporal dementia, 11 semantic dementia [SemD], 4 progressive non-fluent aphasia, 27 Alzheimer's disease, 6 corticobasal degeneration, 9 progressive supranuclear palsy, 13 healthy older controls) were tested using the Social Inference - Minimal subtest of The Awareness of Social Inference Test (TASIT). Subjects watched brief videos depicting sincere or sarcastic communication and answered yes-no questions about the speaker's intended meaning. All groups interpreted Sincere (SIN) items normally, and only the SemD group was impaired on the Simple Sarcasm (SSR) condition. Patients failing the SSR performed more poorly on dynamic emotion recognition tasks and had more neuropsychiatric disturbances, but had better verbal and visuospatial working memory than patients who comprehended sarcasm. Voxel-based morphometry analysis of SSR scores in SPM5 demonstrated that poorer sarcasm comprehension was predicted by smaller volume in bilateral posterior parahippocampi (PHc), temporal poles, and R medial frontal pole (pFWE<0.05). This study provides lesion data suggesting that the PHc may be involved in recognizing a paralinguistic speech profile as abnormal, leading to interpretive processing by the temporal poles and right medial frontal pole that identifies the social context as sarcastic, and recognizes the speaker's paradoxical intentions.