Comparing spectra for standardization

These are spectra of a check sample acquired with and without an option called "compatibility mode", the sample is the same on both spectra, but they are different because in one of them an algorithm is applied, in order that the sample seems to be acquired in another instrument (different bandwidth, and some differences in the wavelength scale). If we compare the spectra without math treatments, we can see small differences (an offset), but we don´t realize if there is a shift in the wavelength scale:

Now we apply a first derivative to these spectra:

Here we can see some shifts in the wavelengths from one instrument to other, but we make a zoom to the areas in yellow to see in more detail.

 

 

Recently I read some advices from Mark Whesterhaus, and one way to decide which type of standardization to apply is to subtract one spectra from the other and to check if it looks to a raw spectrum, to a first derivative or to a second derivative and depending of this you can decide the better standardization algorithm.

This is the spectrum of one raw spectrum (from Master Instrument) substracted from the same sample raw spectrum scanned in Instrument 2 (Host):

The spectrum is very similar to the first derivative spectrum of the sample confirming that there is a instrument shift that must be considered during the standardization.

 

This option has been done exporting the same sample scanned in the same instrument with and without "compatibility mode", which is similar to scan the same sample in 2 different instruments.

Calculating the Regression Coefficients in PCR

This is a simply exercise, indeed to use pls regression, we use this time "pcr" (principal components regression), in "R" we can use  this script:

mod3.pcr<-pcr(Y~X,data=nir.tr1.2dmsc,ncomp=10,validation="LOO")

One of the outputs we get from this calculation is mod3.pcr$coefficients, this output contains the regression coefficients, one for every wavelength. The coefficients change depending of the number of principal components we use. In this case we have chosen a maximum of 10, and we the cross validation we will decide the number of PCs to use.

When we calculate the PCs, we get from the original spectra matrix, two matrices: The score matrix (T) and the loading matrix (P) and they are related with the formula X=T.P^t + E.

We use the T matrix for the calculation of the regression coefficient in the PCR.

Here I do the steps to calculate the regression coefficients in R and I will check finally if they match with the coefficients calculated with the PCR function in the PLS package. I use for this exercise the shootout 2002 data as in previous posts.

Another output from the PCR function is mod3.pcr$scores, which is the T matrix.

This matrix has been calculated in another post:

T.pca<-X1.sg2dmsc_svd_T[,1:10]

It is easy to check that: T.pcr==T.pca  give TRUE for all.

Now we develop in R (with  the T matrix) this formula:

x_hat=(T^t.T)^-1. T^t.Y

T.pca<-X1.sg2dmsc_svd_T[,1:10]     (n.a) matrix

T.pca.t<-t(T.pca)                  (a.n) matrix
Tt.T<-T.pca.t%*%T.pca              (a.a) matrix
Tt.T.inv<-solve(Tt.T)              (a.a) matrix
Tt.T.inv.Tt<-Tt.T.inv%*%T.pca.t    (a.n) matrix
x.hat<-Tt.T.inv.Tt%*%Y             (a.1) matrix

and finally
reg.coef=P.x_hat

reg.coef<-P.pca%*%x.hat            (k.1) matrix

We check if they match:

pcr.coef.10<-as.matrix(mod3.pcr$coefficients[,,10])
pcr.coef.10[1:10,]

> pcr.coef.10[1:10,]

-1.010183  8.223292 14.806965 26.005365 33.214725 20.101344 14.456345 

 9.498212  4.154216 -5.061061 

> reg.coef[1:10,]

-1.010183  8.223292 14.806965 26.005365 33.214725 20.101344 14.456345

 9.498212  4.154216 -5.061061

Projections: more problems

Algebra Lessons: Projections Problem 4_2_1a

 

This is an important Algebra lesson to understand the projections; Chemometrics use them frequently for calculations.

 

Gilbert Strang has a good book: "Introduction to Linear Algebra", with several exercises. From time to time I like to try to solve some of them.

 

 

Searching / Archiving samples in ISI Scan

All the samples in ISI Scan are stored in the "data" folder in the ISI Scan directory. Keeping the Access data base, and  files included into that data folder is important to run the ISI Scan fast and secure.

So it is important to make backups regularly and clean the database at the same time. All this is configures at the System Profile window.

 

In this window, we configure the samples the samples which stay in the database and the samples which are exported or deleted definitively.

Spectra of samples with Lab Value attached are always kept in ISI Scan , so they can be exported as CAL files or used for Monitor purposes.

Read the ISI Scan Help section of System profile for a better understanding of the procedure.
After the Backup, a window appears searching for samples to archive:

Samples are archived in a new folder called Archive inside the Backup folder. Into this Archive folder there are new folders with the name of the products and inside this folders are the NIR files with the spectra, and the ANL or CSV files with the results. So you can export this files to Win ISI to work with them.

 

If the samples stay in the Data folder of ISI Scan and are not yet archived, we can search them clicking with the mouse on Products and selecting "Search". A new window appears where we have filter options, once searched the samples will go to the "Selected Samples" folder.