Modelling complex spectral data (soil) with the resemble package (I)

It is time for a new tutorial working with R, and with one of their packages: "Resemble". We will use soil NIR spectra and we will follow the explanations given by the authors in this vignette:

Modelling complex spectral data with the resemble package (Leonardo Ramirez-Lopez and Alexandre M.J.-C. Wadoux)

From certain time I am interested in the use of NIR spectra to develop models, and to follow this tutorial can help me to understand better how to apply several type of regressions and to see the performance for this complex matrix like soil.

Let´s create a new R Markdown file (.Rmd), and load the three libraries that we will use:

library(tidyverse)
library(resemble)
library(prospectr)
library(magrittr)

Let´s load the data "NIRsoil", and we can see that its class is "dataframe". This data frame combines the spectral matrix (NIRsoil$spec) which contains the predictor variables for every spectrum, with the responses variables (Nt, Ciso and CEC) and another variable which specify if the spectra is used in the training set or the test set. You can read at the vignette details from the authors about what each of these response variable represents and their units.

data("NIRsoil")

Run some code to have details about the number of samples available, the range for each response variable, how many "NA" you have, the wavelength range of the NIR spectrophotometer used,...., and other info that you consider useful before to go into other steps. For example to see the distribution of the Nitrogen:

NIRsoil$Nt %>%
summary()
NIRsoil%>%
ggplot(aes(Nt)) +
geom_histogram()

Min.   1st Qu.  Median   Mean  3rd Qu.   Max.    NA's 

0.200   1.100   1.300   1.766   2.000   8.800     180 

Do the same for the other constituents, changing the "Nt" response variable for the other variables "Ciso" and "CEC".

It is important to check how the response variables correlate between them:

response<- NIRsoil[ , 1:3] %>%
drop_na()
corrplot(cor(response), method = "number")

As we can see there is a high correlation between the parameters.

We can check it in a XY plot for thr "Ciso" and "Nt":

response %>%
ggplot(aes(x = Ciso , y = Nt)) +
geom_point()

This can be a good starting point and we will continue on a new coming post.

Tidymodels: Modeling hotel bookings in R using tidymodels and recipes

In order to use the tidymodels and tidyverse packages to the spectroscopic data we have to familiarize with their functions , so we have to practice with tutorials from other data sets, like in this case with data from hotels bookings.

So lets see Julia working with this data set, and try to practice doing yourself with the available material in tidytuesday and tidimodels websites.

There is a tutorial about Infratec meat spectra with tidymodels (not in video), using PLS , so we will have to look at it in other post.

Max Kuhn - Totally Tidy Tuning Tools

Max Kuhn | parsnip A tidy model interface | RStudio (2019)

Time based validation improvement for pH in cocoa paste

 The best way to check the performance of a calibration is with a new time based validation set. In this case one calibration has been developed to predict pH in cocoa paste. The calibration has been developed choosing the best performance with a cross validation using groups. 

After this the model has been installed in routine and after 2 months, new samples with reference values attached have been collected so we can run the validation to see the statistics. This is the performance:

One of the sample with a high GH is classified as outlier, so we can think that it was a lab error value and we can excluded without a good reason and that must not be done.

Why not to develop again the model with the old training set trying to find a better math-treatment or configuration and validate with this new time based set. The results are surprising with a "None-0-0-1-1" math and 15 PLS terms:

RSQ increase and the sample with the high pH is predicted fine. Take into account that we have a very sort range for pH so we can not expect a high RSQ value.