Individuals with autism spectrum disorders (ASD) are impaired in understanding the emotional undertones of speech, many of which are communicated through prosody. Musical performance also employs a form of prosody to communicate emotion, and the goal of this study was to examine the ability of adolescents with ASD to understand musical emotion. We designed an experiment in which each musical stimulus served as its own control while we varied the emotional expressivity by manipulating timing and amplitude variation. We asked children and adolescents with ASD and matched controls as well as individuals with Williams syndrome (WS) to rate how emotional these excerpts sounded. Results show that children and adolescents with ASD are impaired relative to matched controls and individuals with WS at judging the difference in emotionality among the expressivity levels. Implications for theories of emotion in autism are discussed in light of these findings.