Colored wheat is a valuable resource that is rich in anthocyanins and minerals and thus contributes additional nutritional value to a healthy human diet. However, the effects of nitrogen fertilization on anthocyanin content (AC) and the balance between quality and yield still merit discussion. In this study, blue, purple, and common-colored wheat genotypes were used to investigate three nutrient quality traits, seven processing quality traits, three yield traits and seven grain morphology traits at three nitrogen levels in two years to excavate their possible plasticity under low-nitrogen stress and the tradeoffs among these traits. The highest AC was found in the blue genotypes followed by the purple genotypes. Analysis of variance (ANOVA) showed that AC could be significantly increased by reducing N application, especially in the purple genotypes. Therefore, growing colored wheat with low nitrogen input could allow efficient harvesting of grain with higher AC. However, the other nutrient quality traits and most processing quality traits were observed to decrease under low-nitrogen (LN) stress. Additionally, a correlation analysis indicated that the nutrient quality traits had stable tradeoffs with thousand kernel weight at all N levels because of the significantly negative correlations among them. Therefore, the additive main effect and multiplicative interaction (AMMI) model was used to further identify the most suitable colored genotypes with the best yield potential and also nutrient quality relative characteristics under LN stress. The blue lines Lanmai2999 and purple varieties Zhongkezinuomai 168 were found to be specifically adapted to LN stress with the highest AC values and showed stable performance in the other nutrient quality- and yield-related features. To further investigate the possible mechanism of anthocyanin accumulation in response to reduced N application, the expression of four genes (TaCHS, TaFDR, TaCHI and TaANS) involved in the anthocyanin synthesis pathway was evaluated. All four genes were downregulated under high nitrogen fertilizer application, indicating that anthocyanin synthesis in colored wheat might be inhibited by nitrogen fertilizer. Therefore, this research provided information for optimizing nitrogen fertilizer management in producing colored wheat and also demonstrated that it is efficient and economical to plant colored wheat genotypes in nitrogen-poor areas for use in a healthy human diet, improving the benefits of wheat planting and facilitating nitrogen pollution control.