The mineral concentrations in cereals are important for human health, especially for individuals who consume a cereal subsistence diet. A number of elements, such as zinc, are required within the diet, while some elements are toxic to humans, for example arsenic. In this study we carry out genome-wide association (GWA) mapping of grain concentrations of arsenic, copper, molybdenum and zinc in brown rice using an established rice diversity panel of ∼ 300 accessions and 36.9 k single nucleotide polymorphisms (SNPs). The study was performed across five environments: one field site in Bangladesh, one in China and two in the US, with one of the US sites repeated over two years. GWA mapping on the whole dataset and on separate subpopulations of rice revealed a large number of loci significantly associated with variation in grain arsenic, copper, molybdenum and zinc. Seventeen of these loci were detected in data obtained from grain cultivated in more than one field location, and six co-localise with previously identified quantitative trait loci. Additionally, a number of candidate genes for the uptake or transport of these elements were located near significantly associated SNPs (within 200 kb, the estimated global linkage disequilibrium previously employed in this rice panel). This analysis highlights a number of genomic regions and candidate genes for further analysis as well as the challenges faced when mapping environmentally-variable traits in a highly genetically structured diversity panel.