Fragmentation is generally considered to have negative impacts on widespread outbreeders but impacts on gene flow and diversity in patchy, naturally rare, self-compatible plant species remain unclear. We investigated diversity, gene flow and contemporary pollen-mediated gene immigration in the rare, narrowly distributed endemic shrub Calothamnus quadrifidus ssp. teretifolius. This taxon occurs in an internationally recognized biodiversity hotspot subjected to recent human-induced fragmentation and the condition of the remnants ranges from intact to highly degraded. Using microsatellites, we found that inbreeding, historically low gene flow and significant population differentiation have characterized the genetic system of C. quadrifidus ssp. teretifolius. Inbreeding arises from self-pollination, a small amount of biparental inbreeding and significant correlation of outcross paternity but fecundity was high suggesting populations might have purged their lethals. Paternity analyses show that pollinators can move pollen over degraded and intact habitat but populations in both intact and degraded remnants had few pollen parents per seed parent and low pollen immigration. Genetic diversity did not differ significantly between intact and degraded remnants but there were signs of genetic bottlenecks and reduced diversity in some degraded remnants. Overall, our study suggests human-induced fragmentation has not significantly changed the mating system, or pollen immigration to, remnant populations and therefore genetic connectivity need not be the highest conservation priority. Rather, for rare species adapted to higher levels of inbreeding, conservation efforts may be best directed to managing intact habitats and ecosystem processes.