We dispersively interface an ensemble of 1000 atoms trapped in the evanescent field surrounding a tapered optical nanofiber. This method relies on the azimuthally asymmetric coupling of the ensemble with the evanescent field of an off-resonant probe beam, transmitted through the nanofiber. The resulting birefringence and dispersion are significant; we observe a phase shift per atom of ∼1 mrad at a detuning of 6 times the natural linewidth, corresponding to an effective resonant optical density per atom of 0.027. Moreover, we utilize this strong dispersion to nondestructively determine the number of atoms.