We present a theoretical discovery of an unconventional mechanism of inverse Faraday effect which acts selectively on topological magnetic structures. The effect, topological inverse Faraday effect, is induced by the spin Berry's phase of the magnetic structure when a circularly polarized light is applied. Thus a spin-orbit interaction is not necessary unlike that in the conventional inverse Faraday effect. We demonstrate by numerical simulation that topological inverse Faraday effect realizes ultrafast switching of a magnetic vortex within a switching time of 150 ps without magnetic field.