Diversity of expertise at an individual level can increase intelligence at a collective level-a type of swarm intelligence (SI) popularly known as the 'wisdom of the crowd'. However, this requires independent estimates (rare in the real world owing to the availability of public information) and contradicts people's bias for copying successful individuals. To explain these inconsistencies, 429 people took part in a 'guess the number of sweets' exercise. Guesses made with no public information were diverse, resulting in highly accurate SI. Individuals with access to the previous guess, mean guess or a randomly chosen guess, tended to over-estimate the number of sweets and this undermined SI. However, when people were provided with the current best guess, this prevented very large (inaccurate) guesses, resulting in convergence of guesses towards the true value and accurate SI across a range of group sizes. Thus, contrary to previous work, we show that social influence need not undermine SI, especially where individual decisions are made sequentially and then aggregated. Furthermore, we offer an explanation for why people have a bias to recruit and follow experts in team settings: copying successful individuals can enable accuracy at both the individual and group level, even at small group sizes.