© J Renwick Beattie. Data Scientist. Privacy Policy | Terms and Conditions | External Resource Links | Contact Form

Services Home Osentia Kerus Protein Ageing

Case Note Sections:

Signal Processing

Advanced Signal Processing for Modern Data

The background:


Dr Beattie was lead researcher on a large multi-faculty interdisciplinary consortium investigating potential uses of optical spectroscopy in clinical settings. A consistent recurring ‘feature’ of those datasets was the presence of intense and very variable background unrelated to the portion of the signal that was of interest.

A wide range of cutting edge baseline correction methods were investigated and none were able to reliably and consistently eliminate the background signal. This prevented extraction of clinically useable information from the data.

While it was clear that the intense backgrounds were highly variable and strongly depressing the utility of the data, it was also clear from multivariate analysis of the data that the variations were not random.


Since there was clearly some consistency within the dataset, it must be possible to more consistently correct the problem. Dr Beattie could see the solution in front of his eyes; calculate the baseline correction on the multivariate results. But the software packages he was using were unable to allow manipulation of the type he needed.

Schematic showing the enhanced clarity and insight when data is accuratly processed

Need: reliable processing in noisey data

The solving of this conundrum required a multi pronged approach:


  • Understand the mathematical basis of multivariate analysis, especially PCA
  • Learn how to manipulate data more flexibly
  • Learn how to manipulate a PCA and apply that to new data


A strategy was also devised for testing the approach, which required a number of different experimental approaches to control critical aspects of the baseline and signal. It was essential to understand when the approach was needed and what benefits it may impart


All this had to be implemented in parallel with a full program of research with an emphasis on clinical utility, not interesting and novel methodologies. This meant that each stage of development needed to be applied incrementally and iteratively so that small gains in the signal processing could be exploited for the advancement of the clinical understanding and continuously demonstrate progress and relevance.