We study the statistics of the recurrence times tau between earthquakes above a certain magnitude M in six (one global and five regional) earthquake catalogs. We find that the distribution of the recurrence times strongly depends on the previous recurrence time tau0, such that small and large recurrence times tend to cluster in time. This dependence on the past is reflected in both the conditional mean recurrence time and the conditional mean residual time until the next earthquake, which increase monotonically with tau0. As a consequence, the risk of encountering the next event within a certain time span after the last event depends significantly on the past, an effect that has to be taken into account in any effective earthquake prognosis.