Holographic microscopy combined with forward modeling and inference allows colloidal particles to be characterized and tracked in three dimensions with high precision. However, current models ignore the effects of optical aberrations on hologram formation. We investigate the effects of spherical aberration on the structure of single-particle holograms and on the accuracy of particle characterization. We find that in a typical experimental setup, spherical aberration can result in systematic shifts of about 2% in the inferred refractive index and radius. We show that fitting with a model that accounts for spherical aberration decreases this aberration-dependent error by a factor of two or more, even when the level of spherical aberration in the optical train is unknown. With the new generative model, the inferred parameters are consistent across different levels of aberration, making particle characterization more robust.