Inbreeding depression is most pronounced for traits closely associated with fitness. The traditional explanation is that natural selection eliminates deleterious mutations with additive or dominant effects more effectively than recessive mutations, leading to directional dominance for traits subject to strong directional selection. Here we report the unexpected finding that, in the butterfly Bicyclus anynana, male sterility contributes disproportionately to inbreeding depression for fitness (complete sterility in about half the sons from brother-sister matings), while female fertility is insensitive to inbreeding. The contrast between the sexes for functionally equivalent traits is inconsistent with standard selection arguments, and suggests that trait-specific developmental properties and cryptic selection play crucial roles in shaping genetic architecture. There is evidence that spermatogenesis is less developmentally stable than oogenesis, though the unusually high male fertility load in B. anynana additionally suggests the operation of complex selection maintaining male sterility recessives. Analysis of the precise causes of inbreeding depression will be needed to generate a model that reliably explains variation in directional dominance and reconciles the gap between observed and expected genetic loads carried by populations. This challenging evolutionary puzzle should stimulate work on the occurrence and causes of sex differences in fertility load.