In 1994 the American Academy of Orthopaedic Surgeons established an outcomes initiative. The Musculoskeletal Outcomes Data Evaluation and Management System had two goals: (1) to create validated patient-based functional health questionnaires that would become the gold standard for musculoskeletal research, and (2) to collect data from practicing orthopaedists using these instruments. We assessed the adequacy of the hip and knee portion of the Musculoskeletal Outcomes Data Evaluation and Management System data collection process to learn how to improve these processes. Database elements included demographic and clinical information and health and well-being scales. Only 715 records or 578 patients of the initial 2419 records (30%) had complete baseline and followup information for either hip or knee replacement procedure. Only 17% of the patients who had a knee replacement and 20% of the patients who had a hip replacement returned for followup between 7 and 9 months. The Musculoskeletal Outcomes Data Evaluation and Management System achieved its goal regarding the development of validated questionnaires, but the data collection component did not. The project's implementation and management were flawed. This experience offers the field of orthopaedics valuable lessons that can be applied to future large-scale data collection efforts.