We have observed that cross-polarized backscatter measured by optical coherence tomography of human skin in vivo is surprisingly strong. We identify and give evidence of its main origins: single scattering from nonspherical particles and multiple scattering by particles with sizes much larger than a wavelength. Our findings show that depolarized light scattered by dense large-diameter particles maintains a high degree of temporal coherence and that differential-polarization imaging improves contrast between particles of different sizes.