Osteoporosis is one of the major complications in chronic hepatitis B virus (HBV) and hepatitis C (HCV) infection. However, few studies had examined the relationship between hepatic viral infection with bone loss. Our aim was to investigate the association between hepatic viral infection with bone mineral density (BMD) in a cross-sectional study. Participants who attended the health examinations at the Tri-Service General Hospital (TSGH), Taiwan, were enrolled in the study. Diagnosis of viral hepatitis was confirmed by the serum viral markers of hepatitis B surface antigen (HBsAg) and anti-HCV, and BMD measurement was performed by the bone densitometry. Subjects were divided into four groups by the presence of viral markers. The association between hepatic viral infection and BMD was examined by a multivariate linear regression model. HBV infection was inversely associated with BMD after full adjusting with β values of -0.17 (95% CI: -0.29, -0.05) (p < 0.05). The relationship remained significant in males (β = -0.16, 95% CI = -0.31, -0.01) (p < 0.05). In subjects with body mass index less than 30 HBV infection was associated with reduced BMD (β = -0.16, 95% CI = -0.29, -0.02) (p < 0.05). However, HCV infection was only associated with an increase in BMD in patients with BMI less than 30 (β = 0.17, 95% CI = 0.21, 0.32) (p < 0.05). Chronic HBV infection was significantly associated with reduced BMD in males. The impact of viral hepatitis on bone health deserves further investigation for the potential pathophysiological mechanisms.