Healthy human colon samples were analyzed ex vivo with a multispectral imaging Mueller polarimeter operating from 500 to 700 nm in a backscattering configuration with diffuse light illumination impinging on the innermost tissue layer, the mucosa. The intensity and polarimetric responses were taken on whole tissues first and after progressive exfoliation of the outer layers afterwards. Moreover, these measurements were carried out with two different substrates (one bright and the other dark) successively placed beneath each sample, allowing a reasonably accurate evaluation of the contributions to the overall backscattered light by the various layers. For the shorter investigated wavelengths (500 to 550 nm) the major contribution comes from mucosa and submucosa, while for the longer wavelengths (650 to 700 nm) muscular tissue and fat also contribute significantly. The depolarization has also been studied and is found to be stronger in the red part of the spectrum, mainly due to the highly depolarizing power of the muscular and fat layers.