Quantum mechanics forbids deterministic discrimination among nonorthogonal states. Nonetheless, the capability to distinguish nonorthogonal states unambiguously is an important primitive in quantum information processing. In this work, we experimentally implement generalized measurements in an optical system and demonstrate the first optimal unambiguous discrimination between three nonorthogonal states, with a success rate of 55%, to be compared with the 25% maximum achievable using projective measurements. Furthermore, we present the first realization of unambiguous discrimination between a pure state and a nonorthogonal mixed state.