Object features (e.g. size, shape and orientation) are relevant for recognition and identification, but also for the control of manual actions. Converging evidence suggests a dissociation between the visual systems that mediate object perception and object-directed action. Here we present evidence suggesting that a similar dissociation might exist in the haptic domain. We demonstrate that a haptic variation of a size-contrast illusion influences the perceived size of a target object, but not the degree to which the hand is opened when that object is the target of a grasping movement. This finding is consistent with the view that object perception is "scene-based" and takes into consideration not only the size of the target object but also the sizes of other nearby objects. In contrast, the control of object-directed action is primarily driven by the absolute size of the target object independent of the relative sizes of other objects in the environment, suggesting a "actor-based" frame of reference. The present findings suggest that dissociations between action and perception are not unique to the visual system, but might instead reflect a general organizational principle of sensory processing.