The purpose of this cross-sectional study was to evaluate the quality of life (QOL) of coal dust workers without pneumoconiosis in mainland China. Three hundred five coal dust workers and 200 non-dust workers without pneumoconiosis from five coal mines in Shanxi province were enrolled in this study. The Chinese World Health Organization Quality of Life-brief version (WHOQOL-BREF) questionnaire was used. Socio-demographic, working, and health factors were also collected. Multiple stepwise regression analysis was used to identify significant factors related to the four domain scores of WHOQOL-BREF. All functional domains of the Chinese WHOQOL-BREF were significantly worse in coal dust workers compared to non-dust workers except for psychological health. For the physical domain of QOL, educational level, working hours, and work danger were the significant factors. In the psychological domain, types of job, welfare satisfaction, work danger, hobbies, smoking, one-child family, and marital status were the predictive factors. Working hours, welfare satisfaction, educational level, and birthplace were the predictive factors for the social domain of QOL. Finally, the predictors for the environmental domain of QOL were types of job, working hours, welfare satisfaction, work danger, self-reported social status, smoking, and drinking. Coal dust workers without pneumoconiosis had worse QOL than non-dust workers but their subjective feelings were positive. There were four distinct models for the various domains of QOL. Corresponding health policies could be developed to improve their QOL.