Determining the emission polarization properties of sub-wavelength structures like optical nanoantennas, nanocavities and photonic crystals is important to understand their physical properties and to optimize their use in applications. Recently we have shown that angle-resolved cathodoluminescence imaging spectroscopy (ARCIS), which uses a 30 keV electron beam as an excitation source, is a useful technique to study the far-field properties of such structures. Here we extend the technique with polarization-sensitive angular detection. As proof-of-principle, we experimentally probe the emission polarization properties of three orthogonal dipolar emitters of which the polarization is well-known and find excellent agreement between experiment and theory. We access these dipole orientations by exciting an unstructured gold surface and a ridge nanoantenna with an in-plane dipolar plasmon resonance. The light emission is collected with an aluminum half paraboloid mirror. We show how to take the effect of the paraboloid mirror on the emission polarization into account and how to predict the polarization-filtered pattern if the emission polarization is known. Furthermore, we calculate that by introducing a slit in the beam path the polarization contrast in cathodoluminescence spectroscopy can be strongly enhanced. Finally, we reconstruct the emission polarization from the experimental data and show that from these field patterns we can infer the orientation of the induced dipole moment. The ability to measure the emission polarization, in combination with the sensitivity to the local density of optical states, broad spectral range and high excitation resolution, can be employed to study photonic nanostructures in great detail.