This paper presents a new outlier handling method for volumetric segmentation with three-dimensional (3-D) active shape models. The method is based on a shape metric that is invariant to scaling, rotation and translation by using the ratio of interlandmark distances as a local shape dissimilarity measure. Tolerance intervals for the descriptors are calculated from the training samples and used as a statistical tolerance model to infer the validity of the feature points. A replacement point is then suggested for each outlier based on the tolerance model and the position of the valid points. A geometrically weighted fitness measure is introduced for feature point detection, which limits the presence of outliers and improves the convergence of the proposed segmentation framework. The algorithm is immune to the extremity of the outliers and can handle a highly significant presence of erroneous feature points. The practical value of the technique is validated with 3-D magnetic resonance (MR) segmentation tasks of the carotid artery and myocardial borders of the left ventricle.