Cryptically coloured animals commonly occur in several distinct pattern variants. Such phenotypic diversity may be promoted by frequency-dependent predation, in which more abundant variants are attacked disproportionately often, but the hypothesis has never been explicitly tested. Here we report the first controlled experiment on the effects of visual predators on prey crypticity and phenotypic variance, in which blue jays (Cyanocitta cristata) searched for digital moths on computer monitors. Moth phenotypes evolved via a genetic algorithm in which individuals detected by the jays were much less likely to reproduce. Jays often failed to detect atypical cryptic moths, confirming frequency-dependent selection and suggesting the use of searching images, which enhance the detection of common prey. Over successive generations, the moths evolved to become significantly harder to detect, and they showed significantly greater phenotypic variance than non-selected or frequency-independent selected controls.