Recent work shows that Pockels effect and optics pump-probe measurement could be utilized as a novel method for 511 keV ionizing radiation photon detection for positron emission tomography (PET) which could potentially overcome the inherent physical limitation for coincidence time resolution of around 100 ps (Tao et al 2016 Phys. Med. Biol. 61 7600-22). In this paper, we embrace this observation and introduce a two-crossed-polarizers based setup to achieve similar detection concept, which is a simpler and more compact setup with comparable ionizing radiation detection capability as the setup used in the previously proposed work. We evaluated the performance of our experimental setup with Lithium Niobate (LiNbO3) and Cadmium Telluride (CdTe) detector crystals, and the desired properties of an ideal detector crystal were discussed. The modulation signal induced by 511 keV photons in both LiNbO3 and CdTe can be detected with repeatable signal amplitude using two-crossed-polarizers based method, while CdTe could provide eight times higher detection sensitivity to 511 keV photons than LiNbO3 under the same bias voltage, suggesting high effective Z number and high density properties of CdTe, as well as a shorter carrier lifetime and lower carrier mobility of LiNbO3. In addition, the strength of modulation signal increased linearly with bias voltage before saturation. The modulation signal strength in LiNbO3 continued to increase after 2000 V due to its high resistivity which could reduce the dark current in the detector, while the modulation signal of CdTe with low resistivity tended to be saturated at a bias voltage higher than 1200 V. Therefore, further increasing the bias voltage for detector crystals (especially for LiNbO3) may enhance the modulation strength and improve the detection sensitivity for annihilation photons.