Algorithm development for automated outlier detection and background noise reduction during NIR spectroscopic data processing

David Abookasis, Jerome J. Workman

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


This study describes a hybrid processing algorithm for use during calibration/validation of near-infrared spectroscopic signals based on a spectra cross-correlation and filtering process, combined with a partial-least square regression (PLS) analysis. In the first step of the algorithm, exceptional signals (outliers) are detected and remove based on spectra correlation criteria we have developed. Then, signal filtering based on direct orthogonal signal correction (DOSC) was applied, before being used in the PLS model, to filter out background variance. After outlier screening and DOSC treatment, a PLS calibration model matrix is formed. Once this matrix has been built, it is used to predict the concentration of the unknown samples. Common statistics such as standard error of cross-validation, mean relative error, coefficient of determination, etc. were computed to assess the fitting ability of the algorithm Algorithm performance was tested on several hundred blood samples prepared at different hematocrit and glucose levels using blood materials from thirteen healthy human volunteers. During measurements, these samples were subjected to variations in temperature, flow rate, and sample pathlength. Experimental results highlight the potential, applicability, and effectiveness of the proposed algorithm in terms of low error of prediction, high sensitivity and specificity, and low false negative (Type II error) samples.

Original languageEnglish
Title of host publicationSignal and Data Processing of Small Targets 2011
StatePublished - 2011
EventSignal and Data Processing of Small Targets 2011 - San Diego, CA, United States
Duration: 23 Aug 201125 Aug 2011

Publication series

NameProceedings of SPIE - The International Society for Optical Engineering


ConferenceSignal and Data Processing of Small Targets 2011
Country/TerritoryUnited States
CitySan Diego, CA


  • Calibration and Validation
  • Orthogonal Signal Correction
  • Outlier Detection
  • Partial-Least Square Regression
  • Spectra Correlation

All Science Journal Classification (ASJC) codes

  • Electronic, Optical and Magnetic Materials
  • Condensed Matter Physics
  • Computer Science Applications
  • Applied Mathematics
  • Electrical and Electronic Engineering


Dive into the research topics of 'Algorithm development for automated outlier detection and background noise reduction during NIR spectroscopic data processing'. Together they form a unique fingerprint.

Cite this