curve fitting and parameter estimation in Python

Question

I am currently using Python to compare two different datasets (xDAT and yDAT) that are composed of 240 distance measurements taken over a certain amount of time. However, dataset xDAT is offset by a non-linear amount. This non-linear amount is equal to the width of a time-dependent, dynamic medium, which I call level-A. More specifically xDAT measures from the origin to the top of level-A, whereas yDAT measures from the origin to the bottom of level-A. See following diagram:

In order to compare both curves, I must fist apply a correction to xDAT to make up for its offset (the width of level-A).

As of yet, I have played around with different degrees of numpy.polyfit. I.E:

coefs = np.polynomial.polynomial.polyfit(xDAT, yDAT, 5)
polyEST=[]
for i in range(0,len(x-DAT)):
    polyEST.append(coefs[0] + coefs[1]*xDAT[i] + coefs[2]*pow(xDAT[i],2) + coefs[3]*pow(xDAT[i],3) + coefs[4]*pow(xDAT[i],4) + coefs[5]*pow(xDAT[i],5))

The problem with using this method, is that when I plot polyEST (which is the corrected version of xDAT), the plot still does not match the trend of yDAT and remains offset. Please see the figure below, where xDAT= blue, corrected xDAT=red, and yDAT=green:

Ideally, the corrected xDAT should still remain noisier than the yDAT, but the general oscillation and trend of the curves should match.

I would greatly appreciate help on implementing a different curve-fitting and parameter estimation technique in order to correct for the non-linear offset caused by level-A.

Thank you.

Answer 1

The answer depends on what Level A is. If it is independent, your first line should be something like

coefs = np.polynomial.polynomial.polyfit(numpy.arange(xDAT.size), yDAT-xDAT, 5)

This will give a polyfit of an independent A as drawn, and then the corrected x should be

xDAT+np.polynomial.polynomial.polyval(numpy.arange(xDAT.size),coefs)

If A is dependent on the variables (as it looks to be), you don't want to polyfit, as that only regresses the real part of the oscillation (the "spring" part of a spring-damper system), which is why your corrected_xDat is in phase with xDat instead of yDat. To regress something like that you'll need to use Fourier transforms (which is not my specialty).