Is my model performing as expected? (Part 2)

Really is true when whe say that a picture explain more than a thousand words, and this can be the case when I was trying to explain in the post "Is my model performing as expected? (Part 1)", the decission that a bias should be adjusted looking to the distribution statistics and errors.

But if we overplot the current residuals without adjustment (red dots), and the residuals with the adjustment (blue dots), we can see how the distribution moves into the warning limits.

plot(res~l,main="Residuals",ylim=c(-5*sep,5*sep),
     sub="orange 95% prob / red 99,8% prob",pch=15,col=2,
     xlab="sample position",ylab="residual")
abline(h=0,col="blue")
abline(h=(2*sep),col="orange")
abline(h=(-2*sep),col="orange")
abline(h=(3*sep),col="red")
abline(h=(-3*sep),col="red")
par(new=TRUE)
plot(Table2$res.corr1~l,col=3,ylim=c(-5*sep,5*sep),xlab="sample position",ylab="residual")

Is my model performing as expected? (Part 1)

 Hi all, I am quite busy so I have few time to expend on the blog, anyway I have continue working with R trying to develop functions  in order to check if our models perform as expected or not.

Residual plots and the limits (UAL,UWL,LWL,UAL) we draw on them will help us to take decisions, but developing some functions can help us to see suggestions in order to take good decisions.
So I am trying to works on this.

We always want to compare the results from a Host to a Master, the predicted NIR results with the Lab results,….

In all these predictions we have to provide realistic statistics and not too optimistic, if not we will not understand really how our model performs. Validation statistics, and looking to the residual plots will help us to understand if: our standardization is performing fine, if we have a bias problem or if the samples of the validation should be include in the data set and recalibrate again.

In this case is important to know the RMSEP of our calibration which can be the SECV for example (standard error of cross validation), and compare this error with the RMSEP of the validation, and after this with the SEP (validation error corrected by bias).
Is important to see how the samples are distributed in the residual plot into the warning limits (UWL and LWL) and into the action limits (UAL and LAL), are they distributes randomly?, do they have a bias?, if I correct the bias the distribution becomes random and into limits?,.....There are several questions that if we have the correct answer will help us to improve the model, and to
understand and explain to others the results we obtain.

This is a case where the model performs with a Bias:

Validation Samples  = 9

RMSEP    : 0.62

Bias     : -0.593

SEP      : 0.189

Corr     : 0.991

RSQ      : 0.983

Slope    : 0.928

Intercept: 0.111

RER      : 18.8   Fair

RPD      : 7.02   Excellent

BCL(+/-) : 0.143

***Bias adjustment is recommended***

The residual plot confirms that we have a bias:

Using SEP as std dev the residual distibution is:

  Residuals into 68%   prob (+/- 1SEP)    = 0

  Residuals into 95%   prob (+/- 2SEP)    = 1

  Residuals into 99.5% prob (+/- 3SEP)    = 4

  Residuals outside 99.5% prob (+/- 3SEP) = 5

  Samples outside UAL  = 0

  Samples outside UWL  = 0

  Samples inside   WL  = 1

  Samples outside LWL  = 8

  Samples outside LAL  = 5

With Bias correction the Residual Distribution would be:

  Residuals into 68%   prob (+/- 1SEP) =7

  Residuals into 95%   prob (+/- 2SEP) =9

  Residuals into 99.5% prob (+/- 3SEP) =9

  Residuals out  99.5% prob (> 3SEP)   =0

With the bias correction the statistics are better and confirm that probably a non robust standardization has been done with these two instruments that we are comparing.

This can help us to check other standardizations or decide if we need other algorithms as repeatibility file in the calibration or to mix spectra from both instruments.

RMS calculation in the Diagnostics

If you are use to work with ISI Scan, you can see a noise statistic summary for every cycle (totally 10) of the noise spectra, and two of those statistics are the bias and RMS.

Here I show the statistics for the first four cycles, but we only consider the NIR segment:

 

This post checks how ISI Scan calculates the RMS, and we can see that this RMS value is the RMS corrected by the Bias, so it tells us a measure of the random noise.

I show a simple script showing this:

cycle1<-noise[1,]
cycle2<-noise[2,]
cycle3<-noise[3,]
cycle4<-noise[4,]

 

options(digits=2)
rms1<-sqrt(mean((cycle1)^2)-(rowMeans(cycle1))^2)
rms2<-sqrt(mean((cycle2)^2)-(rowMeans(cycle2))^2)
rms3<-sqrt(mean((cycle3)^2)-(rowMeans(cycle3))^2)
rms4<-sqrt(mean((cycle4)^2)-(rowMeans(cycle4))^2)

> rms1
    1 
0.014 
> rms2
    2 
0.016 
> rms3
    3 
0.011 
> rms4
    4 
0.015 

Looking to the Residual Matrix

The first plot shows in blue color the residual Matrix "E", after developing a principal components calculation with SVD of some samples of wheat flour without any additive (training set).

Additive (ascorbic acid) was added to the flour in certain levels (50, 100, 200 and 500 ppm) to build a validation set. After acquiring the spectra, I reconstruct these spectra with the loading matrix "P" calculates with the SVD using the training samples. First I calculate the scores anf after the reconstructed spectra multiplying the scores with the P transpose matrix.

Once I get the reconstructed spectra, I subtract the reconstructed from the original validation set, and I get the residuals, that I overplot with red color over the blue ones.

I can see that the RMS is higher, and there is some shape different from the random noise of the blue spectra, so the good product model can be tuned to reject this validation set with some RMS cutoff.

Does this shape something from the spectra from the pure additive?

 

 

In green color we can see the spectra of the ascorbic acid, and I convert it into the residuals scale, and where I see certain similarities is in the band at 2250 nm, where all the peaks from the residuals of the validation set, and the ascorbic peak have the same shape.

Anyway this is a simply study about how residuals could help us to determine if the samples can be considered as Good Product, or they are out of specifications and further investigation is needed studying the residuals to determine the cause.

Artificial Check Cells (DS2500 and DA1650)

In order to have a stable check cell over time, manufacturers prepare samples more stable, but at the same time these samples are predicted by a model, and normally this model is not a robust one, so it is sensible to temperature, stabilization of the instrument, laboratory and instrument conditions.

A new generation of instruments like the DS2500 and DA1650 from Foss comes with an artificial Check Sample which contains a cake of Melamine and Talc. The distance between the cake and the window can vary slightly, so when scanning the Check Sample, you don´t have to look to the value of the subsamples for the parameter called Distance, if not to the average value.

The Distance value uses a formula, which contains 5 different properties of the average spectra.

The distance value will change depending of the temperature, so it is important to scan it when the instrument is fully warm up, and also the lab conditions. It is not recommended to keep in the Check Sample History, scans at different conditions.

Check Sample Definition must be redefined after Instrument calibration or lamp replacement.

“Be aware that the artificial check sample cell is just for instrument monitoring. The can indicate that something is going on that needs investigating but seldom alone justify a corrective action” (FOSS advice).