Test accuracy studies assume the existence of a well-defined illness definition and clear-cut diagnostic gold standards or reference standards. However, in clinical reality illness definitions may be vague or a mere description of a set of manifestations, mostly clinical signs and symptoms. This can lead to disagreements among experts about the correct classification of an illness and the adequate reference standard. Using data from a diagnostic accuracy study in carpal tunnel syndrome, we explored the impact of different definitions on the estimated test accuracy and found that estimated test performance characteristics varied considerably depending on the chosen reference standard. In situations without a clear-cut illness definition, randomized controlled trials may be preferable to test accuracy studies for the evaluation of a novel test. These studies do not determine the diagnostic accuracy, but the clinical impact of a novel test on patient management and outcome.