Clinical trials targeting recently elucidated synaptic defects in fragile X syndrome (FXS) will require outcome measures capable of assessing short-term changes in cognitive functioning. Potentially useful measures for FXS were evaluated here in a test-retest setting in males and females with FXS (N = 46). Good reproducibility, determined by an interclass correlation (ICC) or weighted kappa (kappa) of 0.7-0.9 was seen for RBANS List and Story Memory, NEPSY Tower, Woodcock-Johnson Spatial Relations and the commissions score from the Carolina Fragile X Project Continuous Performance Test (CPT). This study demonstrates the feasibility of generating test profiles containing reliability data, ability levels required for test performance, and refusal rates to assist with choice of outcome measures in FXS and other cohorts with cognitive disability.