[FIXED] inverse transform throws error in LSTM prediction

Issue

I am having a LSTM which will predict the value for the next timestep .So my inputs are scaled into the LSTM like below:

test_X = reframed.values
test_X = test_X[:test_X.shape[0], :]
scaler = PowerTransformer()
test_X = scaler.fit_transform(test_X)
test_X = test_X.reshape((test_X.shape[0], 1, test_X.shape[1]))
print(test_X.shape) 

(1, 1, 17)

Then I predict like below :

yhat = model.predict(test_X)

And I have a value for that .

Now I want to inverse scale it so I apply

inv_yhat = scaler.inverse_transform(yhat)

But this throws an error like below :

ValueError: Input data has a different number of features than fitting data. Should have 17, data has 1

Not sure why is this error throwing when the model already predicted the value but just inverse scaling is left.

EDIT :

I use the scalers like this in my training

scaler_x = PowerTransformer()
scaler_y = PowerTransformer()
train_X = scaler_x.fit_transform(train_X)
train_y = scaler_y.fit_transform(train_y.reshape(-1,1))
test_X = scaler_x.fit_transform(test_X)
test_y = scaler_y.fit_transform(test_y.reshape(-1,1))
print(train_X.shape, train_y.shape, test_X.shape, test_y.shape) 



import joblib
joblib.dump(scaler_x, 'scaler_x.gz')
joblib.dump(scaler_y, 'scaler_y.gz')

Then I scale my new fetaures like below :

scaler = joblib.load('scaler.gz')
test_X =scaler.transform(test_X)

still throwing the same error.

Any help is appreciated.

Solution

You're trying to do the inverse transform on your y data when you only applied the transform to the x data. The difference in the number of features (17 for x; 1 for y) is why you get that error

EDIT:

You need to create separate scalers for your x and y data e.g.

scaler_x = PowerTransformer().fit(test_x)
scaler_y = PowerTransformer().fit(test_y) # can be any transformer
scaled_test_x = scaler_x.transform(test_x)
scaled_test_y = scaler_y.transform(test_y)

Then when you want to make a prediction from your fitted model:

yhat = model.predict(test_x)
inv_yhat = scaler_y.inverse_transform(yhat)

Answered By - kynnem