We report on in situ x-ray reflectivity measurements of the surface roughness of supercooled glass forming polymers in a temperature range from 190 to 330 K. The experimentally determined rms roughness has been found to obey the capillary wave model of a single liquid/vapor interface over the entire temperature range. An expression for the surface roughness below the bulk glass transition (T(G) approximately equal to 200 K) is deduced from the viscoelastic theory of surface fluctuations; however, no indication of a frozen-in surface roughness has been observed in the experiment. Additionally, it is shown that precise values of the surface tension of highly viscous liquids in the supercooled state can be determined by x-ray reflectivity.