We compute the lattice thermal conductivity (kappa) of silicon nanowires as a function of temperature by molecular dynamics simulations. In wires with amorphous surfaces kappa may reach values close to that of amorphous silicon and is nearly constant between 200 and 600 K; this behavior is determined by the presence of a majority of nonpropagating vibrational modes. We develop a parameter-free model that accounts for the temperature dependence observed in our simulations and provides a qualitative explanation of recent experiments.