Soil clay regressions: Looking for the better accuracy (part 1)

To have a good traceability in our data bases is important to develop accurate calibrations. We have seen with the Soil LUCAS database that we can filter the data by sample origin (Spain for my case) and after that filter it by land type (I choose for this example “Croplands). The samples are split into training and test set randomly.

 After that we must decide if we choose all the wavelength range (VIS + NIR) or the just the NIR. In this case the calibration is for Clay and different test tell me that the complete range is the best option.

 After this is time to check for the best math treatment trying in this case with 2º SG derivative, 1º SG derivative and SNV+Detrend scatter correction. The last two options gave me better validation statistics than the 2º SG derivative when using a PLS regression

 This is the XY plot for the validation set:

Can we improve the results with another type of regression for these cropland samples? This is what we will see in the next coming posts.

Another question can be: May I use this database to predict samples from another database? These can be samples from a different area in the same country, taken with a different instrument and analysed in a different laboratory. Check all this is important to see the robustness of the calibrations.

GLOSOLAN Soil Spectroscopy Webinar | Interpretation of Mid Infrared Spec...

Great Webinar about the interpretation of Soil Analysis with MIR

Looking to the residuals for the different model updates.

It is important to monitor the residuals (reference - predicted) to check the performance of the models all the way untill we find that our model become robust. In this case we start with a LOCAL model for moisture which does not represent quite well the new samples so the residuals are quite large. Some of these samples were added to the LOCAL database developing a new model (UPG1) which was again evaluated with a new test set (new samples), and we continue doing this for 2 more times.

Upgrades 2 and 3 give the result and  the GH value so we can add to every residual point the GH value to check if the sample is an outlier, and we can see that points far apart than the zero line have a high GH value.

Using Random Forest models in Soil Near Infrared Analysis (part 3)

Once we have tuned the model with the cross validation and the batch best value for "mtry", we can develop the final model we will use for routine and to check the performance with the test set we have leave apart. That is what we will do in the part 4 in the next post.

In this one I show the code for the model and the plot of the importance of every predictor variable (wavelength) in the model. 

I compare the importance scores with the SG second derivative, with the raw Calcite spectrum.

CaCO3_rf_NIRfit <- randomForest(CaCO3 ~., data = CaCO3spcSG_train,
                                importance = TRUE, ntree = 500,
                                mtry = 28)

rfImp <- varImp(CaCO3_rf_NIRfit, scale = FALSE)

matplot(seq(1110, 2488, 2), rfImp, type = "l", ylab = "Importance", 

        xlab = "wavelengths", col = "blue", lwd = 2)
par(new = TRUE)
#Overplot the Calcite spectrum
matplot(seq(1110, 2488, 2), calcite_spectrum_2nm[356:1045, ], type = "l",

        xlab = " ", ylab = " ", yaxt='n', col = "red" )
legend("topleft", # Add legend to plot
        legend = c("Importance Scores", "Calcite spectrum"),
        col = c("Blue", "red"),
        lty = 1)

Using Random Forest models in Soil Near Infrared Analysis (part 2)

In this post I continue from I have left in "Using Random Forest models in Soil Near Infrared Analysis (part 1)", where we developed a Random Forest Model for Carbonate (CaCO3) in soil. Once we have the models and predictions for all the folds, we can plot the actual versus predicted for each fold:

The cross validation can help us to see if we have possible outliers, looking to the plots.

Can we improve the Model?  Yes, why not! Just try to use a batch process to tune for the best hyper-parameters (in this case for the "mtry" argument). Let´s develop a batch sequence from 2 to 30:

cv_tune <- cv_data %>%
crossing(mtry = 2:30)

Developing the models we get these RMSE mean values:

The smallest value is for mtry = 28, and after that one stars increasing.