The objectives of this experiment were 1) to determine the effect of dietary soybean hulls (SBH) and wheat middlings (WM) on body composition, nutrient and energy retention, and the NE of diets and ingredients fed to growing or finishing pigs and 2) to determine if finishing pigs use the energy in SBH and WM more efficiently than growing pigs. Forty growing barrows (initial BW: 25.4 ± 0.7 kg) and 40 finishing barrows (initial BW: 84.8 ± 0.9 kg) were randomly allotted to 5 groups within each stage of growth. Two groups at each stage of growth served as the initial slaughter group. The remaining pigs were randomly assigned to 3 dietary treatments and harvested at the conclusion of the experiment. The basal diet was based on corn and soybean meal and was formulated to be adequate in all nutrients. Two additional diets were formulated by mixing 70% of the basal diet and 30% SBH or 30% WM. In the growing phase, ADG, G:F, and retention of lipids were greater (P < 0.05) for pigs fed the basal diet than for pigs fed the diets containing SBH or WM. Retention of energy was also greater (P < 0.05) for pigs fed the basal diet than for pigs fed the SBH. In the finishing phase, pigs fed the SBH diet tended (P = 0.10) to have a greater ADG than pigs fed the WM diet, and energy retention was greater (P < 0.05) for pigs fed the basal diet than for pigs fed the WM diet. The NE of the basal diet fed to growing pigs was greater (P < 0.01) than the NE of the diets containing SBH or WM, and there was a tendency for a greater (P = 0.05) NE of the basal diet than of the other diets when fed to finishing pigs. The NE of SBH did not differ from the NE of WM in either growing or finishing pigs, and there was no interaction between ingredients and stage of growth on the NE of diets or ingredients. The NE of diets for growing pigs (1,668 kcal/kg) was not different from the NE of diets for finishing pigs (1,823 kcal/kg), and the NE of the diets containing SBH (1,688 kcal/kg) was not different from the NE of the diets containing WM (1,803 kcal/kg). Likewise, the NE of SBH (603 kcal/kg) did not differ from the NE of WM (987 kcal/kg). In conclusion, inclusion of 30% SBH or WM decreases the performance and nutrient retention in growing pigs but has little impact on finishing pigs. The NE of the diets decreases with the inclusion of SBH and WM, but the NE of diets and ingredients is not affected by the BW of pigs. The NE of SBH is not different from the NE of WM.