Herbivores may damage both leaves and reproductive structures, and although such combined damage may affect plant fitness non-additively, this has received little attention. We conducted a 2-year field experiment with a factorial design to examine the effects of simulated leaf (0, 12.5, 25, or 50% of leaf area removed) and inflorescence damage (0 vs. 50% of inflorescences removed) on survival, growth and reproduction in the perennial herb Arabidopsis lyrata. Leaf and inflorescence damage negatively and independently reduced flower, fruit and seed production in the year of damage; leaf damage also reduced rosette size by the end of the first season and flower production in the second year. Leaf damage alone reduced the proportion of flowers forming a fruit and fruit production per plant the second year, but when combined with inflorescence damage no such effect was observed (significant leaf × inflorescence damage interaction). Damage to leaves (sources) caused a greater reduction in future reproduction than did simultaneous damage to leaves and inflorescences (sinks). This demonstrates that a full understanding of the effects of herbivore damage on plant fitness requires that consequences of damage to vegetative and reproductive structures are evaluated over more than 1 year and that non-additive effects are considered.