Intrinsically disordered proteins (IDPs) are notoriously challenging to study both experimentally and computationally. The structure of IDPs cannot be described by a single conformation but must instead be described as an ensemble of interconverting conformations. Atomistic simulations are increasingly used to obtain such IDP conformational ensembles. Here, we have compared the IDP ensembles generated by eight all-atom empirical force fields against primary small-angle X-ray scattering (SAXS) and NMR data. Ensembles obtained with different force fields exhibit marked differences in chain dimensions, hydrogen bonding, and secondary structure content. These differences are unexpectedly large: changing the force field is found to have a stronger effect on secondary structure content than changing the entire peptide sequence. The CHARMM 22* ensemble performs best in this force field comparison: it has the lowest error in chemical shifts and J-couplings and agrees well with the SAXS data. A high population of left-handed α-helix is present in the CHARMM 36 ensemble, which is inconsistent with measured scalar couplings. To eliminate inadequate sampling as a reason for differences between force fields, extensive simulations were carried out (0.964 ms in total); the remaining small sampling uncertainty is shown to be much smaller than the observed differences. Our findings highlight how IDPs, with their rugged energy landscapes, are highly sensitive test systems that are capable of revealing force field deficiencies and, therefore, contributing to force field development.