We study the mean first-passage time τMFP for the barrier crossing of a single massive particle with non-Markovian memory by Langevin simulations in one dimension. In the Markovian limit of short memory time τΓ, the expected Kramers turnover between the overdamped (high-friction) and the inertial (low-friction) limits is recovered. Compared to the Markovian case, we find barrier crossing to be accelerated for intermediate memory time, while for long memory time, barrier crossing is slowed down and τMFP increases with τΓ as a power law τMFP∼τΓ2. Both effects are derived from an asymptotic propagator analysis: while barrier crossing acceleration at intermediate memory can be understood as an effective particle mass reduction, slowing down for long memory is caused by the slow kinetics of energy diffusion. A simple and globally accurate heuristic formula for τMFP in terms of all relevant time scales of the system is presented and used to establish a scaling diagram featuring the Markovian overdamped and the Markovian inertial regimes, as well as the non-Markovian intermediate memory time regime where barrier crossing is accelerated and the non-Markovian long memory time regime where barrier crossing is slowed down.