Most quality control pathways target misfolded proteins to prevent toxic aggregation and neurodegeneration1. Dimerization quality control further improves proteostasis by eliminating complexes of aberrant composition2, but how it detects incorrect subunits remains unknown. Here we provide structural insight into target selection by SCF-FBXL17, a dimerization-quality-control E3 ligase that ubiquitylates and helps to degrade inactive heterodimers of BTB proteins while sparing functional homodimers. We find that SCF-FBXL17 disrupts aberrant BTB dimers that fail to stabilize an intermolecular β-sheet around a highly divergent β-strand of the BTB domain. Complex dissociation allows SCF-FBXL17 to wrap around a single BTB domain, resulting in robust ubiquitylation. SCF-FBXL17 therefore probes both shape and complementarity of BTB domains, a mechanism that is well suited to establish quality control of complex composition for recurrent interaction modules.