Does a behavioral and anatomical division exist between spatial and object working memory? In this article, we explore this question by testing human participants in simple visual working memory tasks. We compared a condition in which there was no location change with conditions in which absolute location change and absolute plus relative location change were manipulated. The results showed that object memory was influenced by memory for relative but not for absolute location information. Furthermore, we demonstrated that relative space can be specified by a salient surrounding box or by distractor objects with no touching surfaces. Verbal memory was not influenced by any type of spatial information. Taken together, these results indicate that memory for "where" influences memory for "what." We propose that there is an asymmetry in memory according to which object memory always contains location information.