We introduce a framework for the reconstruction of the amplitude, phase, and polarization of an optical vector-field using measurements acquired by an imaging device characterized by an integral transform with an unknown spatially variant kernel. By incorporating effective regularization terms, this new approach is able to recover an optical vector-field with respect to an arbitrary representation system, which may be different from the one used for device calibration. In particular, it enables the recovery of an optical vector-field with respect to a Fourier basis, which is shown to yield indicative features of increased scattering associated with tissue abnormalities. We demonstrate the effectiveness of our approach using synthetic holographic images and biological tissue samples in an experimental setting, where the measurements of an optical vector-field are acquired by a multicore fiber endoscope, and observe that indeed the recovered Fourier coefficients are useful in distinguishing healthy tissues from tumors in early stages of oesophageal cancer.