Seed vigor is closely related to direct seeding in rice (Oryza sativa L.). Previous quantitative trait locus (QTL) studies for seed vigor were mainly derived from bi-parental segregating populations and no report from natural populations. In this study, association mapping for seed vigor was performed on a selected sample of 540 rice cultivars (419 from China and 121 from Vietnam). Population structure was estimated on the basis of 262 simple sequence repeat (SSR) markers. Seed vigor was evaluated by root length (RL), shoot length (SL) and shoot dry weight in 2011 and 2012. Abundant phenotypic and genetic diversities were found in the studied population. The population was divided into seven subpopulations, and the levels of linkage disequilibrium (LD) ranged from 10 to 80 cM. We identified 27 marker-trait associations involving 18 SSR markers for three traits. According to phenotypic effects for alleles of the detected QTLs, elite alleles were mined. These elite alleles could be used to design parental combinations and the expected results would be obtained by pyramiding or substituting the elite alleles per QTL (apart from possible epistatic effects). Our results demonstrate that association mapping can complement and enhance previous QTL information for marker-assisted selection and breeding by design.