Copying plot values

 One of the new features of Foss Calibrator update is that we can copy the plot values of an XY plot of predicted vs. actual values, for example, and paste them in as Excel sheet.

Include it with the copy are: sample position and sample number, sample date and time, and predicted and reference values.

We can do the same with the GH and NH plots.

Move reference outliers to a reference outlier sample set (Visual Check)

 Once the clear spectra outliers are remove, we develop the model (In this case N2 in soil) and represent the XY plot (reference values vs. predicted) for the calibration set (blue points) and for the validation set (more than 1000 samples were taken apart from the total set).

Now is our decision to remove some of the samples as reference outliers in the two sets:

Only those that clearly seem reference outliers must be marked, now you can check the statistics, and decide to run the model with other math treatment or calibration strategy (ANN in this case).

Move spectra outliers to a spectra outliers sample set (Visual Check)

 One of the strategies that we have to do when developing a new calibration is to inspect visually the spectra, with the idea to remove or mark the apparently clear outliers.  In the case of Win ISI, if we have a lot of samples it is easy to see them but takes a lot of time find them to delete them.

That point is improve with Foss Calibrator where we can select them with the mouse and mark them as spectral outliers.

There are many reasons for a sample to be an spectral outlier: Instrument was not warmed up, failure in the instrument (lamp or mechanical noise), not a good sample presentation, temperature, or simple that the sample is very different from the rest.

This is the case of soil samples and we start selecting the ones that seem noisy or different from the rest:

We can keep those samples for further detail in a spectra outlier sample set, that at the same time has lab fata in order to validate with them to check if the calibration can extrapolate.

FOSS CALIBRATOR: Tutorial 007

This time we want to test the performance of the models with a new sample set that, I have imported as Validation Set (so it is not divide into, training and validation as in other cases).

First we check if there are any strange spectrum (which is not the case), so we go to Models _ Predict to see how the new samples appear in the XY validation plot versus the samples we have used during the development of the model. A clear bias appear, so we have to improve the model adding this new variability (new company, new batches, samples much more recent than the used in the calibration, new instrument, different laboratory,….).

FOSS CALIBRATOR: Tutorial 006

A calibration is robust if independently of the validation and training sets their predictions are robust as well, so we can try with different sets for training and validation selected randomly, time based, retaining parameter distribution,.....

Foss Calibrator can help quite a lot in this part as you can see in the video.

FOSS CALIBRATOR: Tutorial 005

Time to create the outlier model to predict the Mahalanobis distances in the principal component space.

FOSS CALIBRATOR: Tutorial 004

This is the video number 4 for the Foss Calibrator tutorials in spanish, where a model is developed using the MPLS algorithm. After the model calculation we can see several plots and statistics.

Foss calibrator is very fast for this types of models so we can do several almost at the same time and choose the best one.

Review the statistics and plots trying to finds patterns, outliers,...,etc

FOSS CALIBRATOR: Tutorial - 003

In  the Sample menu, apart to see the spectra we have the option to inspect the samples in a principal components space looking for GH outliers. In this case after changing the default configuration by the one I choose in the previous videos I decide to remove the samples with a GH higher tha 4.00.

These samples are marked as spectral outliers, and we can see them in red color merged with the rest of the spectra, so we can inspect possible reason for those GH values.

We can recalculate the PCs again once these samples are remove, but we will do that in the Model Menu in next videos. This time the idea was to remove what we can consider clear spectral outliers.

In the sample menu we have the option to run PLS to have an idea about how the calibration will work and to check if we have clear reference outliers, taht in the case that the values are not correctly typed we can edit them and change to the correct value.

PCA and PLS will be treated specially in the Models menu, where we use the validation set and we will get the performance statistics in the case of PLS or MPLS Models.

FOSS CALIBRATOR: Importing lab values into a ".nir" file 
FOSS CALIBRATOR: Tutorial 001
FOSS CALIBRATOR: Tutorial 002 

FOSS CALIBRATOR: Tutorial 002

In this second tutorial, we continue looking with more detail to the spectra looking for noise that can be due to the sample presentation or other causes. Unless that noisy area has important information we can remove it for the calculation of outlier models and prediction models.

 Use a higher degree of derivative or lower gaps can help to the detection of noise. 

If there are important information, in the noisy area try to use higher gaps or lower derivative to see if there is an improvement in the spectra shape.

In the case that we are discriminating we have to check if the bands of interest are clearly higher than the noise.

Other tutorials:
FOSS CALIBRATOR: Importing lab values into a ".nir" file 
FOSS CALIBRATOR: Tutorial 001

FOSS CALIBRATOR: Tutorial 001

After importing the "nir" file into the project and link to it the ".csv" file with the lab values, we have generated three logical sets (the total, the training and the validation sets). 

Random split was choose and a warning advice that the validation set is not cover by the training set. We can recalculate or choose other split method, but we continue with these ones.

Without generating any outliers models yet we explore the data into the PCA and PLS space looking for reference or spectral outliers.

Other tutorials:
 FOSS CALIBRATOR: Tutorial 002
 FOSS CALIBRATOR: Importing lab values into a ".nir" file 

FOSS CALIBRATOR: Importing and adding lab values to a ".nir" file

As you see in the video a project is created in Foss Calibrator and a cocoa nir file is imported.

This file can not be divided into a training and test set until a parameter is created and the lab values are imported from a ".csv" file.

Other tutorials:
 FOSS CALIBRATOR: Tutorial 002
 FOSS CALIBRATOR: Tutorial 001

How the number of inputs affects the results in a NIR ANN Calibration

Developing a ANN calibration for ash in meat meal today with Foss Calibrator I realize of the importance of the right selection of inputs in the network. If we take few inputs we can underfit the model and the results could be a bias in the predictions. So prepare a batch mode and don´t be afraid to select a wide range to the limit which in this case is 30. We will get a Heat Map showing the best option for the number of inputs as well for the number of hidden neurons.

 

First I select the default options and I got a bias validating with an independent set. After I make wider the batch for the input layer and the bias disappear with 30 inputs.

Validation with 22 inputs:



Heat mat recommendation with a wider batch:

Validation with 30 inputs:

Foss Calibrator Video (parte 3)

Foss Calibrator Video (parte 2)

Segundo video para familiarizarse con el manejo de Foss Calibrator.

Foss Calibrator Video (parte 1)

Foss Calibrator (quick mPLS overview)

I am starting to use the new software Foss Calibrator, so I will publish some posts about how it works. I use in this case the software for some samples of meat for a viability study of the calibration, and the software improves the split of the sample set into a validation and a calibration set, giving several options like random, time based,...We can choose also if the validation set is into the range of the calibration set, so the model has all the validation samples into the range of the constituent calibration, this way we have quickly the calibration and validation set ready to develop the calibration.

 

For the calibration we have several options for the cross validation (leave one out, using blocks, venetian blinds,......).

We can choose for developing the calibration the options: mPLS, PLS, ANN or LOCAL.I try for this case the mPLS models.

We can select the wavelength range, so we have to look to the spectra to see how if looks and remove noisy part of the spectra, or remove the visible part,.....

The XY plot of Measured vs Predicted shows the calibration and validation samples overlapped and is quite useful for a quick idea of the performance of the model.

 

We have also the plot of the GH distances with the calibration and validation values overlapped:

 

We see the statistics of the model and this time the RMSEP is the total error and the SEP is the error with the bias correction which makes easier to compare the results with other software or literature.

 

 

We can publish the model (calibration and outlier model together) to a folder in our PC and get the ".eqa", ".pca", and ".lib" files to use in Win ISI or load in MOSAIC Network or Solo, and get a report of the calibration.

 

I will continue sharing my experience with Foss Calibrator with the Label "Learning Foss Calibrator"