In this work, we propose a new algorithm for spectral color image segmentation based on the use of a kernel matrix. A cost function for spectral kernel clustering is introduced to measure the correlation between clusters. An efficient multiscale method is presented for accelerating spectral color image segmentation. The multiscale strategy uses the lattice geometry of images to construct an image pyramid whose hierarchy provides a framework for rapidly estimating eigenvectors of normalized kernel matrices. To prevent the boundaries from deteriorating, the image size on the top level of the pyramid is generally required to be around 75 x 75, where the eigenvectors of normalized kernel matrices would be approximately solved by the Nyström method. Within this hierarchical structure, the coarse solution is increasingly propagated to finer levels and is refined using subspace iteration. In addition, to make full use of the abundant color information contained in spectral color images, we propose using spectrum extension to incorporate the geometric features of spectra into similarity measures. Experimental results have shown that the proposed method can perform significantly well in spectral color image segmentation as well as speed up the approximation of the eigenvectors of normalized kernel matrices.