We propose a model to calculate the thermally induced mode loss evolution in the coiled ytterbium doped large mode area (LMA) fiber. The mode loss evolution in the coiled conventional step index LMA 20/400 fiber is investigated. Meanwhile, a model of fiber amplifier considering thermally induced mode loss evolution is established. The higher order mode (HOM) suppression between a co-pumping scheme and counter-pumping scheme under the heat load are compared. The simulation shows that the HOM loss decreases quasi-exponentially with the heat load and the bending radius of the ytterbium doped fiber (YDF) should be optimized according to the heat load to achieve effectively single mode operation. Besides, the counter-pumping fiber amplifier shows much better HOM suppression than the co-pumping fiber amplifier. The results in this paper will provide guidance in the design of novel ytterbium doped LMA fiber and the optimization of the high power single mode fiber amplifier.