Out-of-equilibrium disordered systems may form memories of external driving in a remarkable fashion. The system "remembers" multiple values from a series of training inputs yet "forgets" nearly all of them at long times despite the inputs being continually repeated. Here, learning and forgetting are inseparable aspects of a single process. The memory loss may be prevented by the addition of noise. We identify a class of systems with this behavior, giving as an example a model of non-Brownian suspensions under cyclic shear.