Estimating Multilevel Models on Data Streams

Psychometrika. 2019 Mar;84(1):41-64. doi: 10.1007/s11336-018-09656-z. Epub 2019 Jan 22.

Abstract

Social scientists are often faced with data that have a nested structure: pupils are nested within schools, employees are nested within companies, or repeated measurements are nested within individuals. Nested data are typically analyzed using multilevel models. However, when data sets are extremely large or when new data continuously augment the data set, estimating multilevel models can be challenging: the current algorithms used to fit multilevel models repeatedly revisit all data points and end up consuming much time and computer memory. This is especially troublesome when predictions are needed in real time and observations keep streaming in. We address this problem by introducing the Streaming Expectation Maximization Approximation (SEMA) algorithm for fitting multilevel models online (or "row-by-row"). In an extensive simulation study, we demonstrate the performance of SEMA compared to traditional methods of fitting multilevel models. Next, SEMA is used to analyze an empirical data stream. The accuracy of SEMA is competitive to current state-of-the-art methods while being orders of magnitude faster.

Keywords: Data streams; SEMA; expectation maximization algorithm; machine (online) learning; multilevel models; nested data.

MeSH terms

  • Algorithms*
  • Body Weight
  • Computer Simulation
  • Data Interpretation, Statistical*
  • Female
  • Humans
  • Longitudinal Studies
  • Male
  • Multilevel Analysis*