Auditory learning or experience induces large-scale neural plasticity in not only the auditory cortex but also in the auditory thalamus and midbrain. Such plasticity is guided by acquired sound (sound-specific auditory plasticity). The mechanisms involved in this process have been studied from various approaches and support the presence of a core neural circuit consisting of a subcortico-cortico-subcortical tonotopic loop supplemented by neuromodulatory (e.g., cholinergic) inputs. This circuit has three key functions essential for establishing large-scale and sound-specific plasticity in the auditory cortex, auditory thalamus and auditory midbrain. They include the presence of sound information for guiding the plasticity, the communication between the cortex, thalamus and midbrain for coordinating the plastic changes and the adjustment of the circuit status for augmenting the plasticity. This review begins with an overview of sound-specific auditory plasticity in the central auditory system. It then introduces the core neural circuit which plays an essential role in inducing sound-specific auditory plasticity. Finally, the core neural circuit and its relationship to auditory learning and experience are discussed.