Coarse-grained (CG) models facilitate efficient simulation of complex systems by integrating out the atomic, or fine-grained (FG), degrees of freedom. Systematically derived CG models from FG simulations often attempt to approximate the CG potential of mean force (PMF), an inherently multidimensional and many-body quantity, using additive pairwise contributions. However, they currently lack fundamental principles that enable their extensible use across different thermodynamic state points, i.e., transferability. In this work, we investigate the explicit energy-entropy decomposition of the CG PMF as a means to construct transferable CG models. In particular, despite its high-dimensional nature, we find for liquid systems that the entropic component to the CG PMF can similarly be represented using additive pairwise contributions, which we show is highly coupled to the CG configurational entropy. This approach formally connects the missing entropy that is lost due to the CG representation, i.e., translational, rotational, and vibrational modes associated with the missing degrees of freedom, to the CG entropy. By design, the present framework imparts transferable CG interactions across different temperatures due to the explicit definition of an additive entropic contribution. Furthermore, we demonstrate that transferability across composition state points, such as between bulk liquids and their mixtures, is also achieved by designing combining rules to approximate cross-interactions from bulk CG PMFs. Using the predicted CG model for liquid mixtures, structural correlations of the fitted CG model were found to corroborate a high-fidelity combining rule. Our findings elucidate the physical nature and compact representation of CG entropy and suggest a new approach for overcoming the transferability problem. We expect that this approach will further extend the current view of CG modeling into predictive multiscale modeling.