The principle of parsimony, also known as 'Occam's razor', is a heuristic dictum that is thoroughly familiar to virtually all practitioners of science: Aristotle, Newton, and many others have enunciated it in some form or other. Even though the principle is not difficult to comprehend as a general heuristic guideline, it has proved surprisingly resistant to being put on a rigorous footing - a difficulty that has become more pressing and topical with the 'big data' explosion. We review the significance of Occam's razor in the philosophical and theological writings of William of Ockham, and survey modern developments of parsimony in data science.