Velocity differences in the direct enstrophy cascade of two-dimensional turbulence are correlated with the underlying flow topology. The statistics of the transverse and longitudinal velocity differences are found to be governed by different structures. The wings of the transverse distribution are dominated by strong vortex centers, whereas the tails of the longitudinal differences are dominated by saddles. Viewed in the framework of earlier theoretical work, this result suggests that the transfer of enstrophy to smaller scales is accomplished in regions of the flow dominated by saddles.