Little is known about how well the Centers for Medicare and Medicaid Services' five-star rating system for the overall quality of Medicare Advantage (MA) contracts captures quality of care. Leveraging contract consolidation as a natural experiment to study the association between outcomes and insurer-initiated enrollee shifts to plans with higher-rated contracts, we found that enrollees experiencing a one-star MA rating increase were 20.8 percent less likely to voluntarily leave their plan to enroll in another plan or traditional Medicare. When hospitalized, they were 3.4 percent more likely to use a higher-quality hospital and 2.6 percent less likely to be readmitted within ninety days. Our findings suggest that MA star ratings may capture key domains of an MA plan's quality; however, the differences in outcomes that they capture might not all be clinically meaningful.