We consider the limitations due to noise (e.g., quantum projection noise and photon shot-noise) on the sensitivity of an idealized atomic magnetometer that utilizes spin squeezing induced by a continuous quantum nondemolition measurement. Such a magnetometer measures spin precession of N atomic spins by detecting optical rotation of far-detuned light. We show that for very short measurement times, the optimal sensitivity scales as N(-3/4); if strongly squeezed probe light is used, the Heisenberg limit of N-1 scaling can be achieved. However, if the measurement time exceeds tau(rel)/N(1/2) in the former case, or tau(rel)/N in the latter, where tau(rel) is the spin relaxation time, the scaling becomes N(-1/2), as for a standard shot-noise-limited magnetometer.