SICE: an improved missing data imputation technique

Abstract

In data analytics, missing data is a factor that degrades performance. Incorrect imputation of missing values could lead to a wrong prediction. In this era of big data, when a massive volume of data is generated in every second, and utilization of these data is a major concern to the stakeholders, efficiently handling missing values becomes more important. In this paper, we have proposed a new technique for missing data imputation, which is a hybrid approach of single and multiple imputation techniques. We have proposed an extension of popular Multivariate Imputation by Chained Equation (MICE) algorithm in two variations to impute categorical and numeric data. We have also implemented twelve existing algorithms to impute binary, ordinal, and numeric missing values. We have collected sixty-five thousand real health records from different hospitals and diagnostic centers of Bangladesh, maintaining the privacy of data. We have also collected three public datasets from the UCI Machine Learning Repository, ETH Zurich, and Kaggle. We have compared the performance of our proposed algorithms with existing algorithms using these datasets. Experimental results show that our proposed algorithm achieves 20% higher F-measure for binary data imputation and 11% less error for numeric data imputations than its competitors with similar execution time.

Keywords: Data Analytics; MICE; Missing Data Imputation; Multiple Imputation; Single Imputation.

Fig. 1

Regression lines from two sets of random 100 data taken from 1000 library fine data

Fig. 2

MICE flowchart

Fig. 3

Flowchart of SICE

Fig. 4

Block diagram of the system

Fig. 5

Accuracy and F-measure for four algorithms to impute gender attribute

Fig. 6

Performance comparison of MICE and SICE for additional binary datasets

Fig. 7

Performance of MICE and SICE for ordinal data using PMM and POLYREG

Fig. 8

Comparison of execution time of MICE and SICE to impute UCI car dataset

Fig. 9

Performance of algorithms to predict house prices