Microbial communities and their associated metabolic activity in marine sediments have a profound impact on global biogeochemical cycles. Their composition and structure are attributed to geochemical and physical factors, but finding direct correlations has remained a challenge. Here we show a significant statistical relationship between variation in geochemical composition and prokaryotic community structure within deep-sea sediments. We obtained comprehensive geochemical data from two gravity cores near the hydrothermal vent field Loki's Castle at the Arctic Mid-Ocean Ridge, in the Norwegian-Greenland Sea. Geochemical properties in the rift valley sediments exhibited strong centimeter-scale stratigraphic variability. Microbial populations were profiled by pyrosequencing from 15 sediment horizons (59,364 16S rRNA gene tags), quantitatively assessed by qPCR, and phylogenetically analyzed. Although the same taxa were generally present in all samples, their relative abundances varied substantially among horizons and fluctuated between Bacteria- and Archaea-dominated communities. By independently summarizing covariance structures of the relative abundance data and geochemical data, using principal components analysis, we found a significant correlation between changes in geochemical composition and changes in community structure. Differences in organic carbon and mineralogy shaped the relative abundance of microbial taxa. We used correlations to build hypotheses about energy metabolisms, particularly of the Deep Sea Archaeal Group, specific Deltaproteobacteria, and sediment lineages of potentially anaerobic Marine Group I Archaea. We demonstrate that total prokaryotic community structure can be directly correlated to geochemistry within these sediments, thus enhancing our understanding of biogeochemical cycling and our ability to predict metabolisms of uncultured microbes in deep-sea sediments.