Polymer semiconductors composed of a carbon-based π conjugated backbone have been studied for several decades as active layers of multifarious organic electronic devices. They combine the advantages of the electrical conductivity of metals and semiconductors and the mechanical behavior of plastics, which are going to become one of the futures of modulable electronic materials. The performance of conjugated materials depends both on their chemical structures and the multilevel microstructures in solid states. Despite the great efforts that have been made, they are still far from producing a clear picture among intrinsic molecular structures, microstructures, and device performances. This review summarizes the development of polymer semiconductors in recent decades from the aspects of material design and the related synthetic strategies, multilevel microstructures, processing technologies, and functional applications. The multilevel microstructures of polymer semiconductors are especially emphasized, which plays a decisive role in determining the device performance. The discussion shows the panorama of polymer semiconductors research and sets up a bridge across chemical structures, microstructures, and finally devices performances. Finally, this review discusses the grand challenges and future opportunities for the research and development of polymer semiconductors.