We experimentally analyze collective dynamics of a population of 20 electronic Wien-bridge limit-cycle oscillators with a nonlinear phase-shifting unit in the global feedback loop. With an increase in the coupling strength we first observe formation and then destruction of a synchronous cluster, so that the dependence of the order parameter on the coupling strength is not monotonic. After destruction of the cluster the ensemble remains nevertheless coherent, i.e., it exhibits an oscillatory collective mode (mean field). We show that the system is now in a self-organized quasiperiodic state, predicted in Rosenblum and Pikovsky [Phys. Rev. Lett. 98, 064101 (2007)]. In this state, frequencies of all oscillators are smaller than the frequency of the mean field, so that the oscillators are not locked to the mean field they create and their dynamics is quasiperiodic. Without a nonlinear phase-shifting unit, the system exhibits a standard Kuramoto-like transition to a fully synchronous state. We demonstrate a good correspondence between the experiment and previously developed theory. We also propose a simple measure which characterizes the macroscopic incoherence-coherence transition in a finite-size ensemble.