Streptococcus suis is a zoonotic swine pathogen and a major public health concern in Asia, where it emerged as an important cause of bacterial meningitis in adults. While associated with food-borne transmission in Asia, zoonotic S. suis infections are mainly occupational hazards elsewhere. To identify genomic differences that can explain zoonotic potential, we compared whole genomes of 98 S. suis isolates from human patients and pigs with invasive disease in the Netherlands, and validated our observations with 18 complete and publicly available sequences. Zoonotic isolates have smaller genomes than non-zoonotic isolates, but contain more virulence factors. We identified a zoonotic S. suis clone that diverged from a non-zoonotic clone by means of gene loss, a capsule switch, and acquisition of a two-component signalling system in the late 19th century, when foreign pig breeds were introduced. Our results indicate that zoonotic potential of S. suis results from gene loss, recombination and horizontal gene transfer events.