We aimed to investigate the association of smoking and physical exercise on ventricular function and structure, determined by cardiac magnetic resonance imaging (CMR), in subjects without known cardiovascular diseases. A total of 381 participants (median age 57 years) of the Cooperative Health Research in the Region of Augsburg (KORA) FF4 cohort underwent CMR. The participants' smoking and sporting habits were measured by a questionnaire. Physical inactivity was associated with a reduction of left ventricular ejection fraction (LV-EF), stroke volume, early diastolic peak filling rate and peak ejection rate of the left ventricle as well as right ventricular stroke volume. LV-EF was reduced in subjects with almost no physical activity compared to subjects with regular physical activity (68.4%, 95%CI 66.8-70.1% vs. 70.8%, 95%CI 69.2-72.3%, p < 0,05). Smokers had lower right ventricular end-diastolic volumes (80.6 ml/m², 95%CI 76.7-84.5 ml/m²; never-smokers: 85.5 ml/m², 95%CI 82.6-88.3 ml/m²; p < 0.05) but higher extracellular volume fractions (ECV) and fibrosis volumes (34.3 ml, 95%CI 32.5-36.0 ml, vs. 31.0 ml, 95%CI 29.6-32.3 ml, p < 0.01). We conclude that asymptomatic individuals without known cardiovascular diseases show differences in cardiac function and structure depending on their physical activity and smoking habits. This underlines the importance of prevention and health education.