Moral condemnation of harmful behavior is influenced by both cognitive and affective processes. However, despite much recent research, the proximate source of affect remains unclear. One obvious contender is empathy; simulating the victim's pain could lead one to judge an action as wrong ("outcome aversion"). An alternative, less obvious source is one's own aversion to performing the action itself ("action aversion"). To dissociate these alternatives, we developed a scale that assessed individual aversions to (a) witnessing others experience painful outcomes (e.g., seeing someone fall down stairs); and (b) performing actions that are harmless yet aversive (e.g., stabbing a fellow actor with a fake stage knife). Across 4 experiments, we found that moral condemnation of both first-person and third-party harmful behavior in the context of moral dilemmas is better predicted by one's aversion to action properties than by an affective response to victim suffering. In a fifth experiment, we manipulated both action aversion and the degree of expected suffering across a number of actions and found that both factors make large, independent contributions to moral judgment. Together, these results suggest we may judge others' actions by imagining what it would feel like to perform the action rather than experience the consequences of the action. Accordingly, they provide a counterpoint to a dominant but largely untested assumption that empathy is the key affective response governing moral judgments of harm.