Preparing Spectrum to import into "ChemoSpec"

In this video I show I to prepare a spectrum to import later into ChemoSpec:
The software I use gives the option to export the spectrum as a TXT file.
This file is open with Excel and saved as CSV, to import later with others into ChemoSpec.


For next step, see the post:
 My first "ChemoSpec" spectra

How canola oil is made?

There are NIR applications to determine the quantity of oil can be extracted from the canola seed before entering the production process.
There are intermediate products as the cake where we can also determine parameter characteristics by NIR,  as for example oil (for a second extraction), or other parameters (protein, fiber,....) if the product goes to the feed industry.
Of course there are applications which could be develop  for the final product itself.

This is a nice video to understand the canola oil process:

How olive oil is made & ...Spanish Tapa

It is always positive to understand the process for a certain production in order to understand better the variability we have in our spectra database. That way we will develop better models (qualitative and quantitative).
This is a nice video about the olive oil production:

And why not to add a video from one of my favorite Chefs: José Andres.

He is doing a great effort to promote the Spanish Products in North America.

Olive Oil NIR/VIS Spectra - 001 (ChemoSpec)

I continue with the practicing with ChemSpec, and this time I import seven spectra of olive oil. This time I have been more careful and I have the frequency column as numeric form the CSV file. Once I have the spectra (with and offset):

Take in account the comment form Bryan in the previous post when importing CSV files into ChemSpec.

 I want to go into some details in this post about the visible part of the spectra. So let´s select just this part of the spectra:
plotSpectra(olive, title = "olive oil", which = 1:7,
xlim = c(400,800),yrange = c(0.5,3),offset = 0.3)

Here we can see the bands for the carotenoids (3 peaks in the range between 40-500 nm) and  the band for the chlorophyll  which shifts between 650 and 700 nm. These bands can be change, depending of many circumstances (starting from the quality of the extraction process),  and affects obviously the color of the oil.

These bands change during the oil storage depending of the light conditions and disappear when the oil is refined.

My first "ChemoSpec" spectra

I’ve been this week trying to import some spectra to ChemoSpec (R Package), at the beginning I had problems generating the “csv”  files, and the function getManyCsv did not recognize the files. I did not have properly configured the regional settings in the Control Panel, so I had to configure the column separator correctly, after that the function:

>getManyCsv(gr.crit = c("oliv", "sflw"),gr.cols = c("red3", "dodgerblue4"), freq.unit = "nm",int.unit = "absorbance",

descrip = "oil", out.file = "oil")

generates the oil.RData in my working directory.

I wanted to plot my spectra, but I had problems plotting the data.Hopefully Pof. Bryan Hanson (Author of the Package) told me my mistake (frequency values were as integers), so after changing to numeric I could see finally the spectra. (Thanks Bryan)

The spectra are: one olive oil sample and one sunflower oil sample.

The plot spectra function has the option to add an offset and to see the spectra separated:

plotSpectra(oil, title = "olive/sunflower oil", which = 1:2, offset = 0.3)

 If we remove the offset we will see that the spectra is similar in the NIR region, but now it is important to lokk carefully to the intensity and the wavelength position of the bands to see the diferences, but this will be treated in another post practicing "ChemoSpec".

The noise in the region from 2200nm to 2500 is due that the sample has been analyzed with a glass vial of 8 mm path length so this region must be discarded from the calibration/identification/qualification.

Clearly we can see the differences between the two oil samples. Sunflower oil is refined, so its spectrum is almost flat in the region from 400 to 1100 nm.

Virgin Olive oil has information in that region.

plotSpectra(oil, title = "olive/sunflower oil", which = 1:2)

 If we remove the offset we will see that the spectrais similar in the NIR region, but now it is important the intensity and the wavelength position of the bands to see the diferences, but this will be treated in another post practicing "ChemoSpec".

Reference contamination

In some instruments (working in reflectance) the reference is exposed to moisture, condensation, dust, volatile chemicals,..., and all these  elements contaminate the reference. That is the reason why some instruments tend to protect the reference better in order to avoid these problems.

This is an example: In this case the spectrum of the reference was acquired (32 scans), after that the reference was cleaned carefully.

Let´s subtract the reference before cleaning and after cleaning and look to the patterns, you can get some ideas of the possible contaminants.

Reference subtration spectrum

In some cases are negative spectra similar to some ingredients, fats,, …, which we analyze commonly. It can appear also water vapour signs due to the diference of moisture (water vapour) in the laboratory between the two reference spectra acquisitions (like it seems in this case) .

Of course the scale is amplified quite a lot to see the patterns which can be insignificant for some mayor concentrations, but maybe not for very low concentrations.

Win ISI 4.5 / ISI Scan 4.5 (updates ready)

Win ISI 4.5 and ISI Scan 4.5 are ready for downloading in the Web page http://www.winisi.com/ in the download section.

You can get details in this Web page.

I have installed the new Win ISI version, and now it is possible to export correctly the equation statistics to Excel (bug in the previous version) 

This new version can be installed in Windows 7 (64 b).

ISI Scan 4.5 fix the following bugs from the previous version:

Russian Chemometric Society (Excel Tutorials - 001)

I´m practicing the Excel tutorials from the Russian Chemometric Society, they are really interesting.

Visit their Web page from time to time, it is really interesting.

Estoy practicando los Tutoriales de Excel de la "Russian Chemometric Society" , son realmente interesantes.Podéis sacar mucha información de su página Web.

Should I adjust the slope?

I add a new video “Should I adjust the slope”, where a new part of script is added to the monitor function.  I don´t recommend adjusting the slope, but there are circumstances where it is necessary:

Suppose you have an equation, but not the calibration set from which it was developed, in that case is necessary to adjust the slope and the intercept in some cases, in others just the bias (See the video “Should I adjust the Bias”). We have to take into account that the calibration was developed with the samples scanned in another instrument and with (probably) another reference method, so adjust the slope and intercept is a way to standardize one instrument to the other or one reference method to the other.

To adjust the Bias and the Slope/Intercept, we should have a representative set with samples including all the variance represented in the calibration set. If not any of these adjustments should be done.

If we run the monitor and observe a slope problem in the slope/intercept or bias we have to check the instrument for problems and the reference method as well.

We have to see all the statistics together in order to find an answer.

To see if an adjustment of the slope is necessary, we have to see (with a T-test) if there are significant references that the calculated slope is different from 1.

We have to calculate the residual standard deviation:

, but the predicted values (y hat) are changed by the predicted value corrected with the slope and intercept.

ISO 12099 describes how to get a “t observed” from this RSD, and we compare it with the value obtained from tables for a probability of α =0.05.

I have added these calculations to R script.

These monitor function can be configures the way we prefer. You can take ideas from the ISO12099, or from an interesting article you can download from the Web:
See post: "Some considerations about NIR Spectroscopy"

Improving Monitor statistics with the "UECLs"

New Monitor Video:
Improving Monitor statistics with the "UECLs"

F-test to find UECLs

I have fixed the link to the video "Removing Y outliers from the validation set" and it´s time to see what could be the next step to the function. As we know the RMSEP is the sum of the explained (BIAS) and unexplained error (SEP).

We get also the SEP, so we know the unexplained error, and we can compare bouth in order to see if the Bias is significant, so the adjustment of the Bias is necessary but only temporally until we find the source (if possible) of this explained error. If it is a new source of variance which should be included in the calibration we will develop the model again.

Now I´m working in another function which tell us if the unexplained error (SEP) is out of limits, or what is called the unexplained error confidence limits (UECLs).

This is a  F-test (ratio of two variances): Validation Set and Calibration Set.

In the Calibration Set we have to decide if to choose the SEC (too optimistic) or the SECV (CV error) which is more realistic.

Of course degrees of freedom will be different for bough:

Nv-1 for the validation set

Nc-P-1 for the calibration set

(N =number of samples, P=number of terms).

I will come soon with this, finding if the function is already developed in this case in R (please add your comments), or I have to develop it.

The function will say if a certain SEP value, with a certain number of samples can be accepted if the value exceeds a certain percentage of the SEC or SECV.

Removing "Y" outliers from the "Validation Set"

This is a new video about how to monitor and interpret statistics and graphics for validation.
Removing "Y" outliers from Validation Set

Previous videos about the Monitor function  for validation:
Should I adjust the Bias?
Monitor: Adding "RER" and "RPD" statistics

Other posts with part of the script:
Monitoring some statistics with "R"
Practicing Script with “ R”: Monitor
Improving script_001: “Monitor”
Improving script_002: “Monitor”
Monitor: Removing zero values from the data set.