Numerous studies have shown that physical exercise has positive effects on cognition in healthy adults. Less is known about the effectiveness of exercise interventions in older individuals already suffering from mild cognitive impairment (MCI) or dementia. The aim of this study was to systematically review the evidence from randomized controlled trials (RCTs) of the effects of physical exercise on cognition in older subjects with MCI or dementia. PubMed, Cochrane and DARE databases were systematically searched for RCTs using terms related to cognition and physical exercise. Altogether, 22 trials were found. The studies on older subjects with MCI reported some positive effects of physical exercise on cognition, mainly on global cognition, executive function, attention and delayed recall. However, most studies performed on older subjects with dementia showed no effect of exercise on cognition. The studies had methodological problems in defining dementia/MCI diagnosis, blinding, inadequate sample sizes and not reporting dropouts, compliance or complications. More studies of good quality on older adults with dementia are needed.