In addition to the nucleophile and solvent, the leaving group has a significant influence on SN2 nucleophilic substitution reactions. Its role is frequently discussed with respect to reactivity, but its influence on the reaction dynamics remains unclear. Here, we uncover the influence of the leaving group on the gas-phase dynamics of SN2 reactions in a combined approach of crossed-beam imaging and dynamics simulations. We have studied the reaction F(-) + CH3Cl and compared it to F(-) + CH3I. For the two leaving groups, Cl and I, we find very similar structures and energetics, but the dynamics show qualitatively different features. Simple scaling of the leaving group mass does not explain these differences. Instead, the relevant impact parameters for the reaction mechanisms are found to be crucial and the differences are attributed to the relative orientation of the approaching reactants. This effect occurs on short timescales and may also prevail in solution-phase conditions.