Social decision making involves balancing conflicts between selfishness and pro-sociality. The cognitive processes underlying such decisions are not well understood, with some arguing for a single comparison process, while others argue for dual processes (one intuitive and one deliberative). Here, we propose a way to reconcile these two opposing frameworks. We argue that behavior attributed to intuition can instead be seen as a starting point bias of a sequential sampling model (SSM) process, analogous to a prior in a Bayesian framework. Using mini-dictator games in which subjects make binary decisions about how to allocate money between themselves and another participant, we find that pro-social subjects become more pro-social under time pressure and less pro-social under time delay, while selfish subjects do the opposite. Our findings help reconcile the conflicting results concerning the cognitive processes of social decision making and highlight the importance of modeling the dynamics of the choice process.