Ecological communities are increasingly exposed to multiple chemical and physical stressors, but distinguishing anthropogenic impacts from other environmental drivers remains challenging. Rarely are multiple stressors investigated in replicated studies over large spatial scales (>1000 kms) or supported with manipulations that are necessary to interpret ecological patterns. We measured the composition of sediment infaunal communities in relation to anthropogenic and natural stressors at multiple sites within seven estuaries. We observed increases in the richness and abundance of polychaete worms in heavily modified estuaries with severe metal contamination, but no changes in the diversity or abundance of other taxa. Estuaries in which toxic contaminants were elevated also showed evidence of organic enrichment. We hypothesised that the observed response of polychaetes was not a 'positive' response to toxic contamination or a reduction in biotic competition, but due to high levels of nutrients in heavily modified estuaries driving productivity in the water column and enriching the sediment over large spatial scales. We deployed defaunated field-collected sediments from the surveyed estuaries in a small scale experiment, but observed no effects of sediment characteristics (toxic or enriching). Furthermore, invertebrate recruitment instead reflected the low diversity and abundance observed during field surveys of this relatively 'pristine' estuary. This suggests that differences observed in the survey are not a direct consequence of sediment characteristics (even severe metal contamination) but are related to parameters that covary with estuary modification such as enhanced productivity from nutrient inputs and the diversity of the local species pool. This has implications for the interpretation of diversity measures in large-scale monitoring studies in which the observed patterns may be strongly influenced by many factors that covary with anthropogenic modification.