In order to benefit over the entire frequency range from the injection of squeezed vacuum light at the output of laser gravitational wave detectors, a small bandwidth high finesse cavity is required. In this paper, we investigate the light losses due to the flatness and the roughness of realistic mirrors in a 10 meters-long Fabry-Perot filter cavity. Using measurements of commercial super-polished mirrors, we were able to estimate the cavity round trip losses separating the loss contribution from low and high spatial frequencies. By careful tuning of the cavity g-factor and the incident position of the light on the mirrors, round trip losses due to imperfect mirror surfaces as low as 3 ppm can be achieved in the simulations.