It is important to look at the spectra, before to develop a calibration. Sometimes we trust that all the spectra are fine, so "go ahead", and we won´t get the better results for our statistics. Of course we will keep out clear outliers (an oat spectra placed by mistake in a wheat sample set), or let the software run it like a black box. But software is not such a black box that sometimes could appear, they let us interact choosing different variables, for example the wavelength range.
So look at the spectra, see if there are any trends that are due to changes in the instrument (they start to appear from a certain day), change of sample presentation, and so on.
In this case we want to see the areas more affected by the noise. If it is clear to you keep those areas out from the calibration. Try to find a reason, due to a mechanical problem, ambient temperature,
One way to do it is:
Transform your spectra with the 4th derivative and the smallest segment/gap (normally 1). Display the spectra with their standard deviation. Decide which areas are up of certain "cutoff", and remove those areas from the calibration.