We re-examine the commonly held view that learning and memory necessarily require potentiation of synapses. A simple neuronal model of self-organized learning with no positive reinforcement is presented. The strongest synapses are selected for propagation of activity. Active synaptic connections are temporarily "tagged" and subsequently depressed if the resulting output turns out to be unsuccessful. Thus, all learning occurs by mistakes. The model operates at a highly adaptive state with low activity. Previously stored patterns may be swiftly retrieved when the environment and the demands of the brain change. The combined process of: (i) activity selection by extremal "winner-take-all" dynamics; and (ii) the subsequent weeding out of synapses may be viewed as synaptic Darwinism. We argue that all the features of the model are biologically plausible and discuss our results in light of recent experiments by Fitzsimonds et al. on back-propagation of long-term depression, by Xu et al. on facilitation of long-term depression in the hippocampus by behavioural stress, and by Frey and Morris on synaptic tagging.