Seasonal drought is a major threat to rice production. However, the sensitivity of rice to drought stress (DS) at different growth periods remains unclear. The objective of this study was to reveal the different impacts of DS at the flowering stage on rice physiological traits, grain yield, and quality. Field experiments were conducted with two rice cultivars, Yangliangyou 6 (YLY6) and Hanyou 113 (HY113) under two water treatments (traditional flooding (CK) and DS at flowering stage) in 2013 and 2014. Compared with CK, grain yield (GY) under DS was significantly reduced by 23.2% for YLY6 and 24.0% for HY113 while instantaneous water use efficiency (IWUE) was significantly increased by 39% for YLY6 and 37% for HY113, respectively. All physiological traits were significantly decreased under DS and physiological activities did not revert to normal levels at grain filling stage. There was no significant effect on the appearance and nutritional quality except for the significant increase in chalky kernel and chalkiness under DS. Our data suggest that drought stress at flowering stage has a strong influence on rice physiological traits and yield. Stronger recovery capability contributes to maintaining relatively high grain production, which could be a great target for the breeder in developing drought-tolerant rice cultivars.