A growing body of neuroimaging and neurophysiology studies has demonstrated the motor system's involvement in the observation of actions, but the functional significance of this is still unclear. One hypothesis suggests that the motor system decodes observed actions. This hypothesis predicts that performing a concurrent action should influence the perception of an observed action. We tested this prediction by asking subjects to judge the weight of a box lifted by an actor while the subject either lifted or passively held a light or heavy box. We found that actively lifting a box altered the perceptual judgment; an observed box was judged to be heavier when subjects were lifting the light box, and it was judged to be lighter when they were lifting the heavy box. This result is surprising because previous studies have found facilitating effects of movement on perceptual judgments and facilitating effects of observed actions on movements, but here we found the opposite. We hypothesize that this effect can be understood in terms of overlapping neural systems for motor control and action-understanding if multiple models of possible observed and performed actions are processed.