Optical property modulation induced by ionizing radiation is a promising approach for ultra-fast, lower time jitter detection of photon arrival time. If successful, this method can be utilized in time-of-flight positron emission tomography to achieve a coincidence time resolution approaching 10 ps. In this work, the optical property modulation based method is further developed with focus on a detection setup based on two crossed polarizers. Previous work demonstrated that such an optical setup could be utilized in radiation detection, though its detection sensitivity needed improvement. This work investigates the angle between polarizers and electric field distribution within the detection crystal to understand and improve the detection sensitivity of an optical polarization modulation based method. For this work, cadmium telluride (CdTe) was studied as the detector crystal . The 'magic' angle (i.e. optimal working angle) of the two crossed polarizers based optical setup with CdTe were explored theoretically and experimentally. The experimental results show that the detection sensitivity could be improved by around 10% by determining the appropriate 'magic' angle. We then studied the dependence of detection sensitivity on electric field distribution as well as on the bias voltage across the detector crystal using CdTe crystals. The experimental results show that a smaller electrode on the detector crystal, or a more concentrated electric field distribution could improve detection sensitivity. For CdTe, a detector crystal sample with 2.5 mm × 2.5 mm square electrode has twice the detection sensitivity of a detector crystal with 5 mm × 5 mm square electrode. Increasing the bias voltage before saturation for CdTe could further enhance the modulation strength and thus, the sensitivity. Our investigations demonstrated that by determining the proper working angle of polarizers and bias electrical distribution to the detector, we could improve the sensitivity of the proposed optical setup.