Constructing carbon nanotube (CNT) based heterostructures has proven to be an effective way of improving the microwave absorption (MA) capability of these materials, regardless of whether the heterostructures are located on the inner or outer walls of the CNTs. However, the potential of the two sides of CNTs for constructing efficient MA heterostructures has not been compared, and the underlying mechanism behind this difference has not been determined. Therefore, CNT based heterostructures with Fe2O3 nanoparticles inside (Fe2O3-in-CNTs) and outside (Fe2O3-out-CNTs) of the CNTs were synthesized and characterized. The minimum reflection loss and maximum effective bandwidth of the Fe2O3-in-CNTs are -34.1 dB at 3.0 mm and 5.1 GHz at 2.6 mm, much better than those of the Fe2O3-out-CNTs. Stronger interfacial polarization at the inner surface of the CNTs than at the outer surface was confirmed using off-axis electron holography, which is regarded as the key factor that determines the excellent MA performance of the heterointerface constructed by the inner surface of the CNTs. The attractive potential of the inner surface of CNTs for constructing highly efficient MA heterostructures has, to our knowledge, not been proposed before, the findings of which can shed the light on the approach of developing CNT composited MA materials that have outstanding MA properties.