Complex problem solving (CPS) is an up-and-coming twenty-first century skill that requires test-takers to solve dynamically changing problems, often assessed using computer-based tests. The log data that users produce when interacting with a computer-based test provide valuable information about each individual behavioral action they undertake, but such data are rather difficult to handle from a statistical point of view. This paper addresses this issue by building upon recent research focused on decoding log data and aims to identify homogeneous student profiles with regard to their ability to solve CPS tasks. Therefore, we estimated a discrete two-tier item response theory model, which allowed us to profile units (i.e., students) while taking into account the multidimensionality of the data and the explanatory effect of individual characteristics. The results indicate that: (1) CPS can be thought of as a three-dimensional latent variable; (2) there are ten latent classes of students with homogenous profiles regarding the CPS dimensions; (3) students in the higher latent classes generally demonstrate higher cognitive and non-cognitive performances; (4) some of the latent classes seem to profit from learning-by-doing within tasks, whereas others seem to exhibit the reverse behavior; (5) cognitive and non-cognitive skills, as well as gender and to some extent age, contribute to distinguishing among the latent classes.
Keywords: complex problem solving; discrete two-tier item response theory (IRT) model; log data; profiles of students.