When we reach out for an object with our hand, we transform visual information about the object's position into muscle contractions that will bring our digits to that position. If we reach out with a tool the transformation is different, because the muscle contractions must bring the critical part of the tool to the object, rather than the digits. The difference between the motion of the hand and that of the tool can be quite large, as when moving a computer mouse across a table to bring a cursor to a position on a screen. We examined the responses to unpredictable visual perturbations during such movements. People responded about as quickly to changes in the position of the target when pointing with the mouse as when doing so with their hand. They also responded about as quickly when the cursor was displaced as when the target was displaced. We show that this is not because the visually perceived separation between target and cursor is transformed into a desired displacement of the hand. Our conclusion is that our actions are controlled by the judged positions of the end-effector and the target, even when the former is quite detached from the muscles and joints that are involved in the action.