The Lagrange invariant provides a basic description of an optical imaging system. Many important conclusions can be drawn from it. We discovered that the Lagrange invariant is violated in a self-interference holography system with particular characteristics. With a proof-of-principle system, we proved this violation both theoretically and experimentally. This finding enables future exciting possibilities in optical imaging.