The connectivity of marine populations is often surprisingly lower than predicted by the dispersal capabilities of propagules alone. Estimates of connectivity, moreover, do not always scale with distance and are sometimes counterintuitive. Population connectivity requires more than just the simple exchange of settlers among populations: it also requires the successful establishment and reproduction of exogenous colonizers. Marine organisms often disperse over large spatial scales, encountering very different environments and suffering extremely high levels of post-colonization mortality. Given the growing evidence that such selection pressures often vary over spatial scales that are much smaller than those of dispersal, we argue that selection will bias survival against exogenous colonizers. We call this selection against exogenous colonizers a phenotype-environment mismatch and argue that phenotype-environment mismatches represent an important barrier to connectivity in the sea. Crucially, these mismatches may operate independently of distance and thereby have the potential to explain the counterintuitive patterns of connectivity often seen in marine environments. We discuss how such mismatches might alter our understanding and management of marine populations.