The potential of grazing lands to sequester carbon must be understood to develop effective soil conservation measures and sustain livestock production. Our objective was to evaluate the effects of grazing on soil organic carbon (SOC), total nitrogen (TN), microbial biomass carbon (MBC) in Typical steppe and Desert steppe ecosystems, which are both important grassland resources for animal grazing and ecological conservation in China, and to derive region-specific soil C changes associated with different stocking rates (ungrazed, UG; lightly grazed, LG; moderately grazed, MG; heavily grazed, HG). This study substantiated that significant higher SOC, TN and MBC appeared with the treatment of LG in typical steppe. From 2004 to 2010, grazing treatments increased soil carbon storage in desert steppe, which was partly due to the grazing history. The higher MBC concentration and MBC/SOC suggest a great potential for carbon sequestration in the desert steppe ecosystem. The greater MBC in desert steppe than typical steppe was mainly the result of higher precipitation and temperature, instead of soil substrate. The change of MBC and the strong positive relationships between MBC and SOC indicated that MBC in the soil was a sensitive index to indicate the dynamics of soil organic carbon in both steppes in Inner Mongolia of China.