Robust smoothing of gridded data in one and higher dimensions with missing values

Abstract

A fully automated smoothing procedure for uniformly-sampled datasets is described. The algorithm, based on a penalized least squares method, allows fast smoothing of data in one and higher dimensions by means of the discrete cosine transform. Automatic choice of the amount of smoothing is carried out by minimizing the generalized cross-validation score. An iteratively weighted robust version of the algorithm is proposed to deal with occurrences of missing and outlying values. Simplified Matlab codes with typical examples in one to three dimensions are provided. A complete user-friendly Matlab program is also supplied. The proposed algorithm - very fast, automatic, robust and requiring low storage -provides an efficient smoother for numerous applications in the area of data analysis.

Figure 1

Pseudocode of the automatic robust smoothing algorithm. The variable names and acronyms are described in the text. The complete Matlab code is supplied in the supplemental material (smoothn.m). When robust option is required, minimization of the GCV score is performed at the first robust iterative step only (see 4.c.ii) and the last estimated smoothness parameter (s) is used during the successive steps. This makes the algorithm faster without altering the final results significantly.

Figure 2

Automatic smoothing of two-dimensional data with missing values. A. Noisy data. B. Corrupted data with missing values. C. Smoothed data restored from B. D. Absolute errors between the restored and original data.

Figure 3

Non robust versus robust smoothing. Outliers may bend the smoothed curve (A). A robust smoothing may get rid of this drawback (B).

Figure 4

Automatic smoothing of three-dimensional data.

Figure 5

Automatic smoothing of a noisy cardioid.

Figure 6

Global average land temperature anomaly (ºC) with respect to 1961–1990: smoothed versus original year-averaged data. Year-averaged data are available in http://hadobs.metoffice.com/crutem3/diagnostics/global/nh+sh/annual.

Figure 7

August 2003 surface temperature difference (ºC) with respect to 1961–1990: smoothed (bottom panel) versus original year-averaged (top panel) data. Year-averaged dataset is available in http://hadobs.metoffice.com/hadcrut3/index.html. The smoothed output has been upsampled for a better visualization of the surface temperatures.