Declarative memory is supported by distributed brain networks in which the medial-temporal lobes (MTLs) and pFC serve as important hubs. Identifying the unique and shared contributions of these regions to successful memory performance is an active area of research, and a growing literature suggests that these structures often work together to support declarative memory. Here, we present data from a context-dependent relational memory task in which participants learned that individuals belonged in a single room in each of two buildings. Room assignment was consistent with an underlying contextual rule structure in which male and female participants were assigned to opposite sides of a building and the side assignment switched between buildings. In two experiments, neural correlates of performance on this task were evaluated using multiple neuroimaging tools: diffusion tensor imaging (Experiment 1), magnetic resonance elastography (Experiment 1), and functional MRI (Experiment 2). Structural and functional data from each individual modality provided complementary and consistent evidence that the hippocampus and the adjacent white matter tract (i.e., fornix) supported relational memory, whereas the ventromedial pFC/OFC (vmPFC/OFC) and the white matter tract connecting vmPFC/OFC to MTL (i.e., uncinate fasciculus) supported memory-guided rule use. Together, these data suggest that MTL and pFC structures differentially contribute to and support contextually guided relational memory.