The Casimir-Polder-Lifshitz force felt by an atom near the surface of a substrate is calculated out of thermal equilibrium in terms of the dielectric function of the material and of the atomic polarizability. The new force decays like 1/z3 at large distances (i.e., slower than at equilibrium), exhibits a sizable temperature dependence, and is attractive or repulsive depending on whether the temperature of the substrate is higher or smaller than the one of the environment. Our predictions can be relevant for experiments with ultracold atomic gases. Both dielectric and metal substrates are considered.