The human visual system can extract 3-D shape information of unfamiliar moving objects from their projected transformations. Computational studies of this capacity have established that 3-D shape can be extracted correctly from a brief presentation, provided the moving objects are rigid. The human visual system requires a longer temporal extension, but it can cope with considerable deviations from rigidity. It is shown how the 3-D structure of rigid as well as nonrigid objects can be recovered by maintaining an internal model of the viewed object and modifying it at each instant by the minimal nonrigid change that is sufficient to account for the observed transformation. The results of applying this incremental rigidity scheme to rigid and nonrigid objects in motion are described and compared with human perception.