This study aims to investigate the impact of caregiving on the health status and quality of life (QOL) of primary informal caregivers (PCGs) of elderly care recipients in Hong Kong. A total of 246 PCGs and 492 matched noncaregiver (NCG) controls were identified in a population-based cross-sectional study through random telephone dialing. Their health status and QOL were assessed based on structured questionnaires and Short Form 36 (SF-36) Health Survey. Multiple conditional logistic regression analysis showed that compared with NCGs, PCGs had significantly increased risks for reporting worse health, more doctor visits, anxiety and depression, and weight loss. Female PCGs were more likely to report chronic diseases, symptoms, and insomnia. PCGs, particularly women, had significantly lower scores in all eight domains of SF-36 Health Survey. High caregiver burden score (Zarit Burden Scale) was positively associated with adverse physical and psychological health and poorer QOL. The results indicate that PCGs, particularly women, had an adverse physical and psychological health profile and poorer QOL compared with NCGs.