How do we know where we are looking? A frequent assumption is that the subjective experience of our direction of gaze is assigned to the location in the world that falls on our fovea. However, we find that observers can shift their subjective direction of gaze among different nonfoveal points in an afterimage. Observers were asked to look directly at different corners of a diamond-shaped afterimage. When the requested corner was 3.5° in the periphery, the observer often reported that the image moved away in the direction of the attempted gaze shift. However, when the corner was at 1.75° eccentricity, most reported successfully fixating at the point. Eye-tracking data revealed systematic drift during the subjective fixations on peripheral locations. For example, when observers reported looking directly at a point above the fovea, their eyes were often drifting steadily upwards. We then asked observers to make a saccade from a subjectively fixated, nonfoveal point to another point in the afterimage, 7° directly below their fovea. The observers consistently reported making appropriately diagonal saccades, but the eye movement traces only occasionally followed the perceived oblique direction. These results suggest that the perceived direction of gaze can be assigned flexibly to an attended point near the fovea. This may be how the visual world acquires its stability during fixation of an object, despite the drifts and microsaccades that are normal characteristics of visual fixation.