Recurrent Neural Networks for Multivariate Time Series with Missing Values

Abstract

Multivariate time series data in practical applications, such as health care, geoscience, and biology, are characterized by a variety of missing values. In time series prediction and other related tasks, it has been noted that missing values and their missing patterns are often correlated with the target labels, a.k.a., informative missingness. There is very limited work on exploiting the missing patterns for effective imputation and improving prediction performance. In this paper, we develop novel deep learning models, namely GRU-D, as one of the early attempts. GRU-D is based on Gated Recurrent Unit (GRU), a state-of-the-art recurrent neural network. It takes two representations of missing patterns, i.e., masking and time interval, and effectively incorporates them into a deep model architecture so that it not only captures the long-term temporal dependencies in time series, but also utilizes the missing patterns to achieve better prediction results. Experiments of time series classification tasks on real-world clinical datasets (MIMIC-III, PhysioNet) and synthetic datasets demonstrate that our models achieve state-of-the-art performance and provide useful insights for better understanding and utilization of missing values in time series analysis.

Figure 1

Demonstration of informative missingness on MIMIC-III dataset. The bottom figure shows the missing rate of each input variable. The middle figure shows the absolute values of Pearson correlation coefficients between missing rate of each variable and mortality. The top figure shows the absolute values of Pearson correlation coefficients between missing rate of each variable and each ICD-9 diagnosis category. More details can be found in supplementary information Section S1.

Figure 2

An example of measurement vectors x_t, time stamps s_t, masking m_t, and time interval δ_t.

Figure 3

Graphical illustrations of the original GRU (top-left), the proposed GRU-D (bottom-left), and the whole network architecture (right).

Figure 4

Classification performance on Gesture synthetic datasets with different correlation values.

Figure 5

Plots of input decay γ_xt for all variables (top) and histrograms of hidden state decay weights $W_{{\gamma }_h}$ for 10 variables (bottom) in GRU-D model for predicting mortality on PhysioNet dataset.

Figure 6

Early prediction capacity and model scalability comparisons of GRU-D and other RNN baselines on MIMIC-III dataset.