Caribbean coral reefs are becoming structurally simpler, largely due to human impacts. The consequences of this trend for reef-associated communities are currently unclear, but expected to be profound. Here, we assess whether changes in fish assemblages have been non-random over several decades of declining reef structure. More specifically, we predicted that species that depend exclusively on coral reef habitat (i.e., habitat specialists) should be at a disadvantage compared to those that use a broader array of habitats (i.e., habitat generalists). Analysing 3727 abundance trends of 161 Caribbean reef-fishes, surveyed between 1980 and 2006, we found that the trends of habitat-generalists and habitat-specialists differed markedly. The abundance of specialists started to decline in the mid-1980s, reaching a low of ~60% of the 1980 baseline by the mid-1990s. Both the average and the variation in abundance of specialists have increased since the early 2000s, although the average is still well below the baseline level of 1980. This modest recovery occurred despite no clear evidence of a regional recovery in coral reef habitat quality in the Caribbean during the 2000s. In contrast, the abundance of generalist fishes remained relatively stable over the same three decades. Few specialist species are fished, thus their population declines are most likely linked to habitat degradation. These results mirror the observed trends of replacement of specialists by generalists, observed in terrestrial taxa across the globe. A significant challenge that arises from our findings is now to investigate if, and how, such community-level changes in fish populations affect ecosystem function.