A novel scheme to eliminate the artificial background phase jitter is proposed for measuring the carrier-envelope phase drift of tunable infrared femtosecond pulses from an OPA laser. Different from previous methods, a reference spectral interference measurement is performed, which reveals the artificial phase jitter in the measurement process, in addition to the normal f-to-2f interference measurement between the incident laser pulses and it second harmonic. By analyzing the interference fringes, the accurate CEP fluctuation of the incident pulses is obtained.