The concept of fitness as a measure for a species' success in natural selection is central to the theory of evolution. We here investigate how reproduction rates which are not constant but vary in response to environmental fluctuations, influence a species' prosperity and thereby its fitness. Interestingly, we find that not only larger growth rates but also reduced sensitivities to environmental changes substantially increase the fitness. Thereby, depending on the noise level of the environment, it might be an evolutionary successful strategy to minimize this sensitivity rather than to optimize the reproduction speed. Also for neutral evolution, where species with exactly the same properties compete, variability in the growth rates plays a crucial role. The time for one species to fixate is strongly reduced in the presence of environmental noise. Hence, environmental fluctuations constitute a possible explanation for effective population sizes inferred from genetic data that often are much smaller than the census population size.