We demonstrated characterizing the electric field waveform of multi-terahertz pulses (10 - 50 THz) as vector quantities in the time domain by applying the polarization modulated electro-optic sampling (POMEOS) method. The problem of an ultrabroadband gate pulse was solved by modifying the fitting function in POMEOS, and its validity was confirmed through numerical simulations. High accuracy and precision of approximately 1 mrad with 3 s accumulation were demonstrated. Our method can be applied not only to multi-terahertz polarization measurements for linear response but also to the evaluation of the driving field of intense pulses for nonlinear response or material control.