We model and compare the thermal conductivity of rough semiconductor nanowires (NWs) of Si, Ge, and GaAs for thermoelectric devices. On the basis of full phonon dispersion relations, the effect of NW surface roughness on thermal conductivity is derived from perturbation theory and appears as an efficient way to scatter phonons in Si, Ge, and GaAs NWs with diameter D < 200 nm. For small diameters and large root-mean-square roughness Delta, thermal conductivity is limited by surface asperities and varies quadratically as (D/Delta)(2). At room temperature, our model previously agreed with experimental observations of thermal conductivity down to 2 W m(-1) K(-1) in rough 56 nm Si NWs with Delta = 3 nm. In comparison to Si, we predict here remarkably low thermal conductivity in Ge and GaAs NWs of 0.1 and 0.4 W m(-1) K(-1), respectively, at similar roughness and diameter.