In the resummation of collinear gluons emitted together with active partons from the hadrons in the Drell-Yan process, effects of color entanglement become important when the transverse directions are taken into account. It is then no longer possible to write the cross section as the convolution of two soft correlators and a hard part. We show that the color entanglement introduces additional color factors that must be taken into account in the extraction of transverse momentum-dependent parton distribution functions from azimuthal asymmetries. Examples where such effects matter are the extractions of the double Sivers and double Boer-Mulders asymmetries. Furthermore, we will argue why this color entanglement is a basic ingredient already in the tree-level description of azimuthal asymmetries.