A decline in physical and/or cognitive function is a common feature of aging, and frailty has been shown to be associated with cognitive impairment and dementia. This study aimed to evaluate the association between dynapenia, sarcopenia, and cognitive impairment among community-dwelling older people in Taiwan. Data from the I-Lan Longitudinal Aging Study (ILAS) were retrieved for study. Global cognitive function was assessed by Mini-Mental State Examination (MMSE), whereas the Chinese Version Verbal Learning Test, Boston Naming Test, Verbal Fluency Test, Taylor Complex Figure Test, Digits Backward Test, and Clock Drawing Test were used to assess different domains of cognitive function. Association between sarcopenia and global cognitive function as well as all different dimensions of cognitive function were evaluated. Data from 731 elderly participants (mean age 73.4 ± 5.4 years, 53.8% males) were used for study analysis. The overall prevalence of sarcopenia was 6.8%, which was significantly higher in men (9.3% versus 4.1%, p < 0.05). The mean MMSE score was 23.4 ± 4.4 for all participants, and 10.3% of the study participants were cognitively impaired. Sarcopenia was not significantly associated with global cognitive function (odds ratio [OR] = 1.55, p = 0.317), but global cognitive impairment was significantly associated with low physical performance (OR = 2.31, p = 0.003) and low muscle strength (OR = 2.59, p = 0.011). Nonetheless, sarcopenia was significantly associated with impairment in the verbal fluency test (OR = 3.96, p = 0.006) after adjustment for potential confounders. Dynapenia was significantly associated with cognitive impairment in multiple dimensions and global cognitive function, but sarcopenia was only associated with an impaired verbal fluency test. Reduced muscle strength and/or physical performance related to non-muscle etiology were strongly associated with cognitive impairment. More longitudinal studies are needed.