The human diet is naturally varied and contains not only essential nutrients, but also contains molecules that the body actively excludes or minimizes exposure. Among these molecules are xenosterols, of which plant sterols comprise the greatest exposure risk. These sterols comprise approximately 50% of the total sterols we eat, yet we retain <0.5% of these in our bodies. The bulk of this exclusion takes place in the intestine and the heterodimeric transporters ABCG5 and ABCG8 are key to keeping these xenosterols out of our bodies. In normal humans, pharmacological supplementation with plant sterols (and stanols) has been used to lower cholesterol as these impair intestinal absorption/ re-absorption of this molecule; lowering plasma cholesterol has cardiovascular risk benefits. This review challenges whether this intervention is beneficial and may even be harmful. We summarize the evidence involving humans who have genetic disruption of ABCG5/ABCG8 function, from clinical trial data examining plant sterols and cardiovascular risk, from genetic data affecting normal humans and ABCG5/ABCG8 variations to data obtained using animal models. Accumulation of xenosterols in any significant amount is clearly associated with increased toxicity, and data suggest that at even low levels there may be effects. Importantly, there is also a paucity of data showing cardiovascular end-point benefits with plant sterol/stanol supplementation. The summary of evidence highlights not only caution in recommending such strategies to lower plasma cholesterol, but also in investigating how these xenosterols can affect processes ranging from cardiovascular, endocrine, and neurological function.