X-ray grating interferometry has been highlighted in the last decade as a multi-modal X-ray phase-imaging technique for providing absorption, differential phase, and visibility-contrast images. It has been mainly reported that the visibility contrast in the visibility-contrast image originates from unresolvable random microstructures. In this paper, we show that the visibility contrast is even reduced by a uniform sample with flat surfaces due to the so-called "beam-hardening effect", which has to be taken into account when X-rays with a continuous spectrum is used. We drive a criterion for determining whether the beam-hardening effect occurs or not, and propose a method for correcting the effect of beam hardening on a visibility-contrast image.