Starting from the Kramers equation for the phase-space dynamics of the N-body probability distribution, we derive a dynamical density functional theory (DDFT) for colloidal fluids including the effects of inertia and hydrodynamic interactions (HI). We compare the resulting theory to extensive Langevin dynamics simulations for both hard rod systems and three-dimensional hard sphere systems with radially symmetric external potentials. As well as demonstrating the accuracy of the new DDFT, by comparing with previous DDFTs which neglect inertia, HI, or both, we also scrutinize the significance of including these effects. Close to local equilibrium we derive a continuum equation from the microscopic dynamics which is a generalized Navier-Stokes-like equation with additional non-local terms governing the effects of HI. For the overdamped limit we recover analogues of existing configuration-space DDFTs but with a novel diffusion tensor.