Plants produce a diversity of secondary metabolites (SMs) to protect them from generalist herbivores. On the other hand, specialist herbivores use SMs for host plant recognition, feeding and oviposition cues, and even sequester SMs for their own defense. Therefore, plants are assumed to face an evolutionary dilemma stemming from the contrasting effects of generalist and specialist herbivores on SMs. To test this hypothesis, bioassays were performed with F2 hybrids from Jacobaea species segregating for their pyrrolizidine alkaloids (PAs), using a specialist flea beetle (Longitarsus jacobaeae) and a generalist slug (Deroceras invadens). Our study demonstrated that while slug feeding damage was negatively correlated with the concentration of total PAs and that of senecionine-like PAs, flea beetle feeding damage was not affected by PAs. It was positively correlated though, with leaf fresh weight. The generalist slug was deterred by senecionine-like PAs but the specialist flea beetle was adapted to PAs in its host plant. Testing other herbivores in the same plant system, it was observed that the egg number of the specialist cinnabar moth was positively correlated with jacobine-like PAs, while the silver damage of generalist thrips was negatively correlated with senecionine- and jacobine-like PAs, and the pupae number of generalist leaf miner was negatively correlated with otosenine-like PAs. Therefore, while the specialist herbivores showed no correlation whatsoever with PA concentration, the generalist herbivores all showed a negative correlation with at least one type of PA. We concluded that the generalist herbivores were deterred by different structural groups of PAs while the specialist herbivores were attracted or adapted to PAs in its host plants.