Measuring individual differences in statistical learning: Current pitfalls and possible solutions

Abstract

Most research in statistical learning (SL) has focused on the mean success rates of participants in detecting statistical contingencies at a group level. In recent years, however, researchers have shown increased interest in individual abilities in SL, either to predict other cognitive capacities or as a tool for understanding the mechanism underlying SL. Most if not all of this research enterprise has employed SL tasks that were originally designed for group-level studies. We argue that from an individual difference perspective, such tasks are psychometrically weak, and sometimes even flawed. In particular, the existing SL tasks have three major shortcomings: (1) the number of trials in the test phase is often too small (or, there is extensive repetition of the same targets throughout the test); (2) a large proportion of the sample performs at chance level, so that most of the data points reflect noise; and (3) the test items following familiarization are all of the same type and an identical level of difficulty. These factors lead to high measurement error, inevitably resulting in low reliability, and thereby doubtful validity. Here we present a novel method specifically designed for the measurement of individual differences in visual SL. The novel task we offer displays substantially superior psychometric properties. We report data regarding the reliability of the task and discuss the importance of the implementation of such tasks in future research.

Keywords: Individual differences; Psychometrics; Statistical learning.

Figure 1

example for shapes for standard VSL tasks.

Figure 2

Distribution of scores for two individuals that differ in their probability to identify triplets in the test phase (p=0.6 and p=0.8), in tests with 8, 16 and 32 trials, over a simulation of 1000 iterations. As the number of trials increases, the overlap between the distributions decreases and the test better discriminates between the two individuals.

Figure 3

Performance histogram of n=76 in a VSL task, from Siegelman & Frost, 2015. The red line depicts the individual chance level threshold - number of correct trials needed for a given individual to show learning.

Figure 4

Item Response Function for a hypothetical item. The x-axis represents the ability in the measured construct, and the y-axis depicts the expected probability to answer correctly on this specific item. The center square highlights regions where the item is informative and discriminative, while the adjacent rectangles marks regions of the distribution in which the item is not informative.

Figure 5

Example of three trials from the test (left to right): (1) a 4-forced-choice pattern recognition trial with triplets, (2) a 2-forced-choice pattern recognition trial with pairs, and (3) a pattern completion trial of triplet. Instructions were originally presented in Hebrew.

Figure 6

Distribution of scores in the first session. The black dashed line shows the group chance-level (success in 16.67 trials), and the solid red line shows the individual chance-level (success in 23 trials or more).

Figure 7

Test-retest for the new VSL task.