Inferences during discourse comprehension are needed to connect subsequent sentences to each other. Previous studies have implicated the dorso-medial prefrontal cortex (dmPFC) as important for successful coherence building. The present study addressed the question of whether the dmPFC activation could be modulated by the task instructions or whether it was a function of stimulus properties. In a graded coherence judgment task, participants rated on a 4-point scale the closeness of the pragmatic connection between subsequent sentences. 75% of the sentence pairs had previously been identified as incoherent, so that some creativity was needed for finding a pragmatic connection. Of particular interest were the incoherent sentence pairs rated as somewhat related, because this novel condition allowed to separate stimulus properties from the inference success. Compared to the unrelated sentence pairs, this condition elicited increased activation in fronto-parietal regions related to executive functions, reflecting the difficulty of the search for coherence. Compared to coherent sentence pairs, activations in the bilateral fusiform gyri were found. This suggests a role for visual and spatial situation model building during inferencing. Most importantly, the time course of the signal change in the fronto-medial cortex confirmed that all rating conditions engaged this brain region to an equal degree, and that the process extended throughout the relatively long decision period. These results confirm that the dmPFC activation can be modulated by task instructions and that it reflects non-automatic cognitive processes, subserving the integration between external stimulation and idiosyncratic response criteria.