Understanding how to control the nucleation and growth rates is crucial for designing nanoparticles with specific sizes and shapes. In this study, we show that the nucleation and growth rates are correlated with the thermodynamics of metal-ligand/solvent binding for the pre-reduction complex and the surface of the nanoparticle, respectively. To obtain these correlations, we measured the nucleation and growth rates by in situ small angle X-ray scattering during the synthesis of colloidal Pd nanoparticles in the presence of trioctylphosphine in solvents of varying coordinating ability. The results show that the nucleation rate decreased, while the growth rate increased in the following order, toluene, piperidine, 3,4-lutidine and pyridine, leading to a large increase in the final nanoparticle size (from 1.4 nm in toluene to 5.0 nm in pyridine). Using density functional theory (DFT), complemented by 31P nuclear magnetic resonance and X-ray absorption spectroscopy, we calculated the reduction Gibbs free energies of the solvent-dependent dominant pre-reduction complex and the solvent-nanoparticle binding energy. The results indicate that lower nucleation rates originate from solvent coordination which stabilizes the pre-reduction complex and increases its reduction free energy. At the same time, DFT calculations suggest that the solvent coordination affects the effective capping of the surface where stronger binding solvents slow the nanoparticle growth by lowering the number of active sites (not already bound by trioctylphosphine). The findings represent a promising advancement towards understanding the microscopic connection between the metal-ligand thermodynamic interactions and the kinetics of nucleation and growth to control the size of colloidal metal nanoparticles.