During conversation, it is necessary to keep track of what others can and cannot understand. Previous research has focused largely on understanding the time course along which knowledge about interlocutors influences language comprehension/production rather than the cognitive process by which interlocutors take each other's perspective. In addition, most work has looked at the effects of knowledge about a speaker on a listener's comprehension, and not on the possible effects of other listeners on a participant's comprehension process. In the current study, we introduce a novel joint comprehension paradigm that addresses the cognitive processes underlying perspective taking during language comprehension. Specifically, we show that participants who understand a language stimulus, but are simultaneously aware that someone sitting next to them does not understand the same stimulus, show an electrophysiological marker of semantic integration difficulty (i.e., an N400-effect). Crucially, in a second group of participants, we demonstrate that presenting exactly the same sentences to the participant alone (i.e. without a co-listener) results in no N400-effect. Our results suggest that (1) information about co-listeners as well as the speaker affect language comprehension, and (2) the cognitive process by which we understand what others comprehend mirrors our own language comprehension processes.