Hyperspectral stimulated Raman scattering (SRS) imaging has rapidly become an emerging tool for high content analyses of cell and tissue systems. The label-free nature of SRS imaging combined with its chemical specificity allows in situ and in vivo biochemical quantification at submicrometer resolution without sectioning and staining. Current hyperspectral SRS data analysis methods are based on either linear unmixing or multivariate analysis, which are not sensitive to small spectral variations and often provide obscure information on the cell composition. Here, we demonstrate a spectral phasor analysis method that allows fast and reliable cellular organelle segmentation of mammalian cells, without any a priori knowledge of their composition or basis spectra. We further show that, in combination with a branch-bound algorithm for optimal selection of a few wavenumbers, spectral phasor analysis provides a robust solution to label-free single cell analysis.