Brain atrophy may be a useful surrogate marker of axonal loss and disease progression in multiple sclerosis (MS). Several studies have suggested that inflammatory disease activity is a risk factor for atrophy in the early stages of the disease, but may become less important later in the disease course. We aimed to investigate the relationships between atrophy and active inflammation at different stages of the disease course using brain volume measurements from magnetic resonance imaging (MRI) in patients with both relapsing-remitting (RR) (n=95) and secondary progressive (SP) (n=76) MS. Conventional dual echo and three-dimensional magnetization-prepared rapid-acquisition gradient echo imaging were performed in all patients and in 31 healthy controls. Supratentorial and infratentorial brain, and lateral ventricular volumes were determined using modern design stereology. Patients with SP MS had smaller supratentorial (p=0.003) and infratentorial brain volumes (p=0.0003), and larger lateral ventricles (p=0.02) than patients with RR MS. RR MS patients with T(1)-enhancing lesions had smaller supratentorial (p=0.02) and infratentorial (p=0.002) brain volumes and larger ventricles (p=0.002) than those without enhancing lesions. SP MS patients with enhancing lesions also had significantly larger lateral ventricles (p=0.03). Categorical analysis showed that more RR MS patients with enhancing lesions had smaller supratentorial brain (p=0.005), or larger lateral ventricular (p=0.028) volumes, and more SP MS patients with enhancing lesions had increased lateral ventricle volumes (p=0.013), than patients without enhancements. The number of enhancing lesions was significantly correlated with lateral ventricular volumes in both RR MS (r=0.39, p=0.0001) and SP MS (r=0.46, p<0.0001). Our data shows that the presence of active inflammation on a single MRI in the course of RR and SP MS, is associated with a higher risk and higher level of brain atrophy. These findings emphasise the important long-term relationship between inflammation and atrophy in MS and provide additional support for the strategy of early anti-inflammatory treatment to protect tissue integrity.