More about Mahalanobis distance in R

There are several Mahalanobis distance post in this blog, and this post show a new way to find outliers with a library in R called "mvoutlier".

 

Mahalanobis ellipses can only be shown in 2 dimensions with a cutoff value as we have seen, so we show the maps of scores 2 by 2 for the different combinations of PCs, like in this case for PC1 and PC2 and we can mark the outliers in the plot by the identify function: 

In this case I mark some of the samples out of the Mahalanobis distance cutoff. Anyway the Mahalanobis distance is univariate and in this case where we have a certain number of PCs, we have to see not just a map of two of them or all at the same time, we need a unique Mahalanobis distance value and to check if that value is over or into the cutoff value that we assign.

 

For that reason we use the Moutlier function of the "chemometrics" package and show a real Mahalanobis outlier plot which can be Robust or Classical:

 

We can see the classical plot and identify the samples over the cutoff:

 

We can see the list of all the distances in the output list for the function. I will continue with more options to check the Mahalanobis distances in the next post.

Mahalanobis in the PC space (removing redundant) - 2

We have seeing this plot in previous posts, but it is a good occasion to see them again if yo have read the previous post. The function Moutlier from the package chemometrics, have the option to see the outliers,, in the PC space with the normal covariance matrix and the robust covariance matrix. A line is defined for the cutoff value for a certain chi-square distribution, and the samples out are the outliers. Anyway in this example I am considering just two principal components, but when using more PCs, we give just one value for all of them, and it means that maybe a sample is an outlier for one of the PCs, but for the general computation this sample is fine.

A more conservative approach is to put a bigger distance for the cutoff, or a warning an an action cutoff.

In this plot we see the distance respect to the firts two PCs, and we see the same two outliers than in the previous post (four in the case of the Robust Covariance Matrix).

 

But if we consider the PC2 and PC3·, there are not outliers, and all the samples are bellow the cutoff:

This are sample very with certain physical properties, which make them sensitive to be outliers in the first PC, but not in the rest of the PC score maps.

If somebody wants the file to follow the tutorial, just let me know by mail and I will send it to you. The script is on my Github page

 

The chemometric package is the R companion to the book "Introduction to Multivariate Statistical Analysis in Chemometrics" written by K. Varmuza and P. Filzmoser (2009)

 

Mahalanobis in PC space (removing redundat)

In a previous post, we used "prospectr", with the duplex function to select a training set (30 samples), in order to spend less money in lab analysis. This way, we remove redundants spectra, and the selected spectra are well dispersing all around the PC space. Redundant were discarded according to its Neighborhood Mahalanobis distance.

In this plot, we can see the selected samples (red ones), and a Mahalanobis ellipse, considering all the samples with the "drawMahal" function from the package "chemometrics".

If we consider just the red samples (the 30 samples for the training set), the center of the population will be different that with all the samples (156), so we have to calculate the PCs again (their orientation will change), and we can draw a new Mahalanobis  ellipse (97.5 quantile).

As we can see in the plots,  removing the redundantswill give more weigth to the extreme samples, to be retained, and not be consider as outliers.