Quantum ghost image technique utilizing position or momentum correlations between entangled photons can realize nonlocal reconstruction of the image of an object. In this work, based on polarization entanglement, we experimentally demonstrate quantum ghost imaging of vector images by using a geometric phase object. We also provide a corresponding theoretical analysis. Additionally, we offer a geometrical optics path explanation of ghost imaging for vector fields. The proposed strategy offers new insights into the fundamental development of ghost imaging and also holds great promise for developing complex structured ghost imaging techniques. Our work expanding the principle of ghost imaging to spatially varying vector beams will lead to interesting developments of this field.