A bifurcating system subject to multiplicative noise can display on-off intermittency. Using a canonical example, we investigate the extreme sensitivity of the intermittent behavior to the nature of the noise. Through a perturbative expansion and numerical studies of the probability density function of the unstable mode, we show that intermittency is controlled by the ratio between the departure from onset and the value of the noise spectrum at zero frequency. Reducing the noise spectrum at zero frequency shrinks the intermittency regime drastically. This effect also modifies the distribution of the duration that the system spends in the off phase. Mechanisms and applications to more complex bifurcating systems are discussed.