The past two decades have seen a tremendous surge in scientific interest in the extent to which certain types of training-be it aerobic, athletic, musical, video game, or brain trainer-can result in general enhancements in cognitive function. While there are certainly active debates regarding the results in these domains, what is perhaps more pressing is the fact that key aspects of methodology remain unsettled. Here we discuss a few of these areas including expectation effects, test-retest effects, the size of the cognitive test battery, the selection of control groups, group assignment methods, difficulties in comparing results across studies, and in interpreting null results. Specifically, our goal is to highlight points of contention as well as areas where the most commonly utilized methods could be improved upon. Furthermore, because each of the sub-areas above (aerobic training through brain training) share strong similarities in goal, theoretical framework, and experimental approach, we seek to discuss these issues from a general perspective that considers each as members of the same broad "training" domain.