On the basis of paleological evidence, it has been suggested that biological evolution need not necessarily be characterized by gradual change. Rather, evolutionary history may display saltatory periods of rapid speciation alternating with periods of relative quiescence, the whole dynamic being called punctuated equilibria. The empirical evidence that has been presented in support of this hypothesis has been the object of a vigorous dispute. Mathematical investigations of complex models of biological evolution that contain random elements have demonstrated that these systems can display saltatory behavior. In this paper we address a more abstract question: can saltations occur in the evolution of very simple, deterministic mathematical systems that function in a constant environment? The answer appears to be yes. Saltations appear as a natural dynamical behavior in the evolution of simplistic information processing networks. We stress that these networks do not constitute a model of biological evolution. However, the appearance of saltations in such simple systems suggests that their appearance in a process as complex as biological evolution is not surprising.