In this paper, we use single-trial analysis of electroencephalography (EEG) to ascertain the cortical origins of response time variability in a rapid serial visual presentation (RSVP) task. We extract spatial components that maximally discriminate between target and distractor stimulus conditions over specific time windows between stimulus onset and the time of a motor response. We then compute the peak latency of this differential activity on a trial-by-trial basis, and correlate this with response time. We find, for our nine participants, that the majority of the latency is introduced by component activity which begins far-frontally 200 ms prior to the response and proceeds to become parietally distributed near the time of response. This activity is consistent with the hypothesis that cortical networks involved in generating the late positive complexes may be the origins of the observed response time variability in rapid discrimination of visual objects.