Phylogenetic relationships in the Cetacean suborder Mysticeti (baleen whales) have recently been the focus of increased attention. Here, we examine the evolutionary history of this group by comparing genealogies derived from Y chromosome and mitochondrial DNA sequences. We generated topologies based on paternally and maternally inherited characters for males from nine baleen whale species, including representatives of three families (Balaenidae, Eschrichtiidae, and Balaenopteridae) and four genera (Balaena, Eschrichtius, Balaenoptera, and Megaptera). Divergence among species was fifteen times greater for mtDNA than for Y-specific DNA. Both mtDNA and yDNA topologies revealed the family Balaenopteridae to be paraphyletic, but this relationship was neither strongly supported nor consistent across phylogenetic analysis methodologies. Humpback and fin whales, representing different genera, were reciprocally monophyletic sister species according to mtDNA. Although the monophyly of fin whales decayed for yDNA, a close relationship between fin and humpback whales was retained in yDNA trees. The paraphyly of fin whales and the long branch leading to humpback whales for the yDNA marker may suggest life history differences between these species. Specifically, male humpback whales showed higher than average divergence from other baleen whales at yDNA, although not at mtDNA, suggesting a potential for smaller effective population sizes among male humpbacks on an evolutionary timescale. The observation that those species that have been found to hybridize in nature (blue/fin and blue/humpback) do not reveal evidence for paraphyly for either maternal or paternal markers suggests that introgressive hybridization has not historically been extensive and thus may not represent a substantial source of phylogenetic error for Mysticeti.