[FIXED] Persistent time shift after cross correlation

Issue

I have two data sets each containing data on the x and y positions of an object. The data is sampled with different sensors. One of the data sets is sampled at 125 Hz the other at 500Hz. I have resampled the 125Hz data to 500Hz and I wish to align the data. The first image shows the data before time aligning (after interpolating 125Hz->500Hz) No time shift The second shows after the time shift calculated from the cross correlation (x top plot, y bottom plot): Time shift after cross correlation

Original data: https://drive.google.com/drive/folders/1shiw8HZmcDydLYanR0nlvoy3iKYt7Ij9?usp=sharing

This is how I calculate the correlation and time shift. (Part of a larger class so I just posted the content - don't mind e.g. 'i in range(3)')

from scipy.signal import correlate

    def cross_correlation(self):
        cross_cor_vec = [
            correlate(
                self.tar500.iloc[:, i],
                self.act125.iloc[:, i],
                mode="full",
            )
            for i in range(3)
        ]
        return cross_cor_vec


    def get_time_shift(self):
        cross_cor_vec = self.cross_correlation()
        normalization_factors = [
            np.sqrt(
                np.sum(self.tar500.iloc[:, i] ** 2)
                * np.sum(self.act125.iloc[:, i] ** 2)
            )
            for i in range(3)
        ]
        correlation_coefficients = [
            cross_cor_vec[i] / normalization_factors[i] for i in range(3)
        ]
        # Find the max correlation among the correlation coefficients.
        corr_dict = {
            np.argmax(correlation_coefficients[i]): np.max(correlation_coefficients[i])
            for i in range(3)
        }
        index_max_corr = max(zip(corr_dict.values(), corr_dict.keys()))[1]
        time_shift = index_max_corr - len(self.act125.x - 1)
        self.tar500.drop(
            self.tar500.index[:time_shift], inplace=True
        )
        self.tar500.reset_index(drop=True, inplace=True)
        self.tar500.drop(
            self.tar500.index[len(self.act125) :], inplace=True
        )

As you can see from the figure a clear time shift is still present even after cross correlation. I have also checked the data against dynamic time warping - and from a purely visual inspection it aligns the data far better.

What am I missing here?

Tried using scipy.signal correlate and timeshifting using np.argmax()

UPDATED: After using Matt Timmermans answer the trajectories are now aligned: updated trajectories

Solution

Your signals have a large DC offset, so you're not correlating two little sine wave segments, you're correlating two little sine wave segments on big (negative) pedestals.

You think you're correlating

__/\  __   *  _____/\  ___
    \/               \/

But you're really correlating

_____          _____   __              __
     |        |          |            |
     |__/\  __|      *   |_____/\  ___|
          \/                     \/

To get the correlation you want, you need to remove the DC components. You would normally apply a high-pass filter with limited support, cut off the part of the filtered signals at the start/end that get messed up, and then apply the correlation.

Unfortunately, you don't have enough signal to apply a high-pass filter that is long enough to pass the interesting part. If you can't get more samples at the start and end, then you can extend it by replicating the average of the first few/last few samples.

Answered By - Matt Timmermans