An increasing amount of evidence supports a crucial role for the anterior temporal lobe (ATL) in semantic processing. Critically, a selective disruption of the functional connectivity between left and right ATLs in patients with chronic aphasic stroke has been illustrated. The aim of the current study was to evaluate the consequences that lesions on the ATL have on the neurocognitive network supporting semantic cognition. Unlike previous work, in this magnetoencephalography study we selected a group of patients with small lesions centered on the left anteroventral temporal lobe before surgery. We then used an effective connectivity method (i.e., dynamic causal modeling) to investigate the consequences that these lesions have on the functional interactions within the network. This approach allowed us to evaluate the directionality of the causal interactions among brain regions and their associated connectivity strengths. Behaviorally, we found that semantic processing was altered when patients were compared with a strictly matched group of controls. Dynamic causal modeling for event related responses revealed that picture naming was associated with a bilateral frontotemporal network, encompassing feedforward and feedback connections. Comparison of specific network parameters between groups revealed that patients displayed selective network adjustments. Specifically, backward connectivity from anterior to posterior temporal lobe was decreased in the ipsilesional hemisphere, whereas it was enhanced in the contralesional hemisphere. These results reinforce the relevance of ATL in semantic memory, as well as its amodal organization, and highlight the role of feedback connections in enabling the integration of the semantic information.