Theories of time perception typically assume that some sort of memory represents time intervals. This memory component is typically underdeveloped in theories of time perception. Following earlier work that suggested that representations of different time intervals contaminate each other (Grondin, 2005; Jazayeri & Shadlen, 2010; Jones & Wearden, 2004), an experiment was conducted in which subjects had to alternate in reproducing two intervals. In two conditions of the experiment, the duration of one of the intervals changed over the experiment, forcing subjects to adjust their representation of that interval, while keeping the other constant. The results show that the adjustment of one interval carried over to the other interval, indicating that subjects were not able to completely separate the two representations. We propose a temporal reference memory that is based on existing memory models (Anderson, 1990). Our model assumes that the representation of an interval is based on a pool of recent experiences. In a series of simulations, we show that our pool model fits the data, while two alternative models that have previously been proposed do not.