[FIXED] How to implement LSTM in tensorflow v1 from pandas dataframe

Issue

I've tried following tutorials on implementing this but I keep getting dimension errors on the LSTM layer.

ValueError: Input 0 of layer LSTM is incompatible with the layer: expected ndim=3, found ndim=2. Full shape received: [None, 2]

import random
import numpy as np
import tensorflow as tf
from tensorflow import feature_column as fc
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, DenseFeatures, Reshape
from sklearn.model_selection import train_test_split

def df_to_dataset(features, target, batch_size=32):
    return tf.data.Dataset.from_tensor_slices((dict(features), target)).batch(batch_size)

# Reset randomization seeds
np.random.seed(0)
tf.random.set_random_seed(0)
random.seed(0)

# Assume 'frame' to be a dataframe with 3 columns: 'optimal_long_log_return', 'optimal_short_log_return' (independent variables) and 'equilibrium_log_return' (dependent variable)
X = frame[['optimal_long_log_return', 'optimal_short_log_return']][:-1]
Y = frame['equilibrium_log_return'].shift(-1)[:-1]
X_train, _X, y_train, _y = train_test_split(X, Y, test_size=0.5, shuffle=False, random_state=1)
X_validation, X_test, y_validation, y_test = train_test_split(_X, _y, test_size=0.5, shuffle=False, random_state=1)

train = df_to_dataset(X_train, y_train)
validation = df_to_dataset(X_validation, y_validation)
test = df_to_dataset(X_test, y_test)

feature_columns = [fc.numeric_column('optimal_long_log_return'), fc.numeric_column('optimal_short_log_return')]

model = Sequential()
model.add(DenseFeatures(feature_columns, name='Metadata'))
model.add(LSTM(256, name='LSTM'))
model.add(Dense(1, name='Output'))
model.compile(loss='logcosh', metrics=['mean_absolute_percentage_error'], optimizer='Adam')
model.fit(train, epochs=10, validation_data=validation, verbose=1)
loss, accuracy = model.evaluate(test, verbose=0)
print(f'Target Error: {accuracy}%')

After seeing this issue elsewhere I've tried setting input_shape=(None, *X_train.shape), input_shape=X_train.shape, neither works. I also tried inserting a Reshape layer model.add(Reshape(X_train.shape)) before the LSTM layer and it fixed the issue but I got another issue in its place:

InvalidArgumentError: Input to reshape is a tensor with 64 values, but the requested shape has 8000

...and I'm not even sure adding the Reshape layer is doing what I think it is doing. After all, why would reshaping the data to its own shape fix things? Something is happening with my data that I just don't understand.

Also, I'm using this for time series analysis (stock returns), so I would think that the LSTM model should be stateful and temporal. Would I need to move the timestamp index into its own column in the pandas database before converting to a tensor?

Unfortunately I'm obligated to use tensorflow v1.15 as this is being developed on the QuantConnect platform and they presumably won't be updating the library any time soon.

EDIT: I've made a bit of progress by using TimeseriesGenerator, but now I'm getting the following error (which returns no results on Google):

KeyError: 'No key found for either mapped or original key. Mapped Key: []; Original Key: []'

Code below (I'm sure I'm using the input_shape arguments incorrectly):

train = TimeseriesGenerator(X_train, y_train, 1, batch_size=batch_size)
validation = TimeseriesGenerator(X_validation, y_validation, 1, batch_size=batch_size)
test = TimeseriesGenerator(X_test, y_test, 1, batch_size=batch_size)

model = Sequential(name='Expected Equilibrium Log Return')
model.add(LSTM(256, name='LSTM', stateful=True, batch_input_shape=(1, batch_size, X_train.shape[1]), input_shape=(1, X_train.shape[1])))
model.add(Dense(1, name='Output'))
model.compile(loss='logcosh', metrics=['mean_absolute_percentage_error'], optimizer='Adam', sample_weight_mode='temporal')
print(model.summary())
model.fit_generator(train, epochs=10, validation_data=validation, verbose=1)
loss, accuracy = model.evaluate_generator(test, verbose=0)
print(f'Model Accuracy: {accuracy}')

Solution

Turns out this specific issue relates to a patch that Quantconnect made to pandas dataframes which interfered with the older version of tensorflow/keras.

Answered By - SnakeWasTheNameTheyGaveMe