Universal distribution of saliencies for pruning in layered neural networks

Int J Neural Syst. 1997 Oct-Dec;8(5-6):489-98. doi: 10.1142/s0129065797000471.


A better understanding of pruning methods based on a ranking of weights according to their saliency in a trained network requires further information on the statistical properties of such saliencies. We focus on two-layer networks with either a linear or nonlinear output unit, and obtain analytic expressions for the distribution of saliencies and their logarithms. Our results reveal unexpected universal properties of the log-saliency distribution and suggest a novel algorithm for saliency-based weight ranking that avoids the numerical cost of second derivative evaluations.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Algorithms
  • Artificial Intelligence*
  • Linear Models
  • Neural Networks, Computer*
  • Nonlinear Dynamics