Short-chain fatty acids (SCFAs) have been regarded as the excellent carbon source of wastewater biological nutrient removal, and sludge alkaline (pH 10) fermentation has been reported to achieve highly efficient SCFAs production. In this study, the underlying mechanisms for the improved SCFAs production at pH 10 were investigated by using 454 pyrosequencing and fluorescent in situ hybridization (FISH) to analyze the microbial community structures in sludge fermentation reactors. It was found that sludge fermentation at pH 10 increased the abundances of Pseudomonas sp. and Alcaligenes sp., which were able to excrete extracellular proteases and depolymerases, and thus enhanced the hydrolysis of insoluble sludge protein and polyhydroxyalkanoates (PHA). Meanwhile, the abundance of acid-producing bacteria (such as Clostridium sp.) in the reactor of pH 10 was also higher than that of uncontrolled pH, which benefited the acidification of soluble organic substrates. Further study indicated that sludge fermentation at pH 10 significantly decreased the number of methanogenic archaea, resulting in lower SCFAs consumption and lower methane production. Therefore, anaerobic sludge fermentation under alkaline conditions increased the abundances of bacteria involved in sludge hydrolysis and acidification, and decreased the abundance of methanogenic archaea, which favored the competition of bacteria over methanogens and resulted in the efficient production of SCFAs.