Caribbean coral reefs have transformed into algal-dominated habitats over recent decades, but the mechanisms of change are unresolved due to a lack of quantitative ecological data before large-scale human impacts. To understand the role of reduced herbivory in recent coral declines, we produce a high-resolution 3,000 year record of reef accretion rate and herbivore (parrotfish and urchin) abundance from the analysis of sediments and fish, coral and urchin subfossils within cores from Caribbean Panama. At each site, declines in accretion rates and parrotfish abundance were initiated in the prehistorical or historical period. Statistical tests of direct cause and effect relationships using convergent cross mapping reveal that accretion rates are driven by parrotfish abundance (but not vice versa) but are not affected by total urchin abundance. These results confirm the critical role of parrotfish in maintaining coral-dominated reef habitat and the urgent need for restoration of parrotfish populations to enable reef persistence.