We consider a classical and possibly driven composite system X⊗Y weakly coupled to a Markovian thermal reservoir R so that an unambiguous stochastic thermodynamics ensues for X⊗Y. This setup can be equivalently seen as a system X strongly coupled to a non-Markovian reservoir Y⊗R. We demonstrate that only in the limit where the dynamics of Y is much faster than X, our unambiguous expressions for thermodynamic quantities, such as heat, entropy, or internal energy, are equivalent to the strong coupling expressions recently obtained in the literature using the Hamiltonian of mean force. By doing so, we also significantly extend these results by formulating them at the level of instantaneous rates and by allowing for time-dependent couplings between X and its environment. Away from the limit where Y evolves much faster than X, previous approaches fail to reproduce the correct results from the original unambiguous formulation, as we illustrate numerically for an underdamped Brownian particle coupled strongly to a non-Markovian reservoir.