In this study, we addressed how the particular context of stimulus congruency influences audiovisual interactions. We combined an audiovisual congruency task with a proportion-of-congruency manipulation. In Experiment 1, we demonstrated that the perceived duration of a visual stimulus is modulated by the actual duration of a synchronously presented auditory stimulus. In the following experiments, we demonstrated that this crossmodal congruency effect is modulated by the proportion of congruent trials between (Exp. 2) and within (Exp. 4) blocks. In particular, the crossmodal congruency effect was reduced in the context with a high proportion of incongruent trials. This effect was attributed to changes in participants' control set as a function of the congruency context, with greater control applied in the context where the majority of the trials were incongruent. These data contribute to the ongoing debate concerning crossmodal interactions and attentional processes. In sum, context can provide a powerful cue for selective attention to modulate the interaction between stimuli from different sensory modalities.