Collecting response times using Amazon Mechanical Turk and Adobe Flash

Abstract

Crowdsourcing systems like Amazon's Mechanical Turk (AMT) allow data to be collected from a large sample of people in a short amount of time. This use has garnered considerable interest from behavioral scientists. So far, most experiments conducted on AMT have focused on survey-type instruments because of difficulties inherent in running many experimental paradigms over the Internet. This study investigated the viability of presenting stimuli and collecting response times using Adobe Flash to run ActionScript 3 code in conjunction with AMT. First, the timing properties of Adobe Flash were investigated using a phototransistor and two desktop computers running under several conditions mimicking those that may be present in research using AMT. This experiment revealed some strengths and weaknesses of the timing capabilities of this method. Next, a flanker task and a lexical decision task implemented in Adobe Flash were administered to participants recruited with AMT. The expected effects in these tasks were replicated. Power analyses were conducted to describe the number of participants needed to replicate these effects. A questionnaire was used to investigate previously undescribed computer use habits of 100 participants on AMT. We conclude that a Flash program in conjunction with AMT can be successfully used for running many experimental paradigms that rely on response times, although experimenters must understand the limitations of the method.

Figure 1

Histograms of durations of inter-stimulus intervals for each condition, as measured by the phototransistor and by the ActionScript code. X-axis is the measured duration in ms.

Figure 2

Durations of intervals recorded by ActionScript for each Turker, ordered by variance.

Figure 3

Left: Reprinted from Nieuwenhuis et al. (2006; Experiment 1). Mean RTs and error rates reported in Nieuwenhuis et al. (2006; Experiment 1). Right: Mean RTs and error rates for each trial category found in the AMT data. Error bars represent standard errors of the mean.

Figure 4

Statistical power of three-way interaction as a function of sample size and significance criterion.

Figure 5

Statistical power of two-way interaction as a function of sample size and significance criterion.