Crosslinking-mass spectrometry (XL-MS) serves to identify interaction sites between proteins. Numerous search engines for crosslink identification exist, but lack of ground truth samples containing known crosslinks has precluded their systematic validation. Here we report on XL-MS data arising from measuring synthetic peptide libraries that provide the unique benefit of knowing which identified crosslinks are true and which are false. The data are analysed with the most frequently used search engines and the results filtered to an estimated false discovery rate of 5%. We find that the actual false crosslink identification rates range from 2.4 to 32%, depending on the analysis strategy employed. Furthermore, the use of MS-cleavable crosslinkers does not reduce the false discovery rate compared to non-cleavable crosslinkers. We anticipate that the datasets acquired during this research will further drive optimisation and development of XL-MS search engines, thereby advancing our understanding of vital biological interactions.