[FIXED] How to apply scipy.signal.filtfilt() on incomplete data

Issue

I want to plot incomplete data (some values are None). In addition I want to apply a butter function on the dataset and show both graphs, incomplete and smoothened. The filter function seems to not work with incomplete data.

Data File: data.csv

import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
from scipy import signal

data = np.genfromtxt('data.csv', delimiter = ',')
df = pd.DataFrame(data)
df.set_index(0, inplace = True)

b, a = signal.butter(5, 0.1)
y = signal.filtfilt(b,a, df[1].values)
df2 = pd.DataFrame(y, index=df.index)

df.plot()
df2.plot()

plt.show()

Solution

The documentation page does not mention anything related to NaN. You may have to first remove the NaN from your list of values. Here is a way to do it using Numpy isnan function:

y = signal.filtfilt(b, a, df[1].values[~np.isnan(df[1].values)])
df2 = pd.DataFrame(y, index=df.index[~np.isnan(df[1].values)])

Answered By - Patol75