[FIXED] Problem in LSTM train-test split in time series data

Issue

I am trying to make a train set and test set with my csv file to train an LSTM. The csv file looks like this:

        datetime      invno          inkw             outkw    Total    wind_spd        temp    pres             ts
2021-12-01  00:00:00    1       0.0                    0.0      0.0         4.6         -0.9    1007.7      1638284400.0
2021-12-01  00:00:00    4       0.0                    0.0      0.0,        4.6         -0.9    1007.7      1638284400.0
2021-12-01  00:00:00    2       0.0                    0.0      0.0,        4.6         -0.9    1007.7      1638284400.0
2021-12-01  00:00:00    3       0.0                    0.0      0.0,        4.6         -0.9    1007.7      1638284400.0
2021-12-01  00:00:00    5       0.0                    0.0      0.0,        4.6         -0.9    1007.7      1638284400.0
2021-12-01  01:00:00    1       0.0                    0.0      0.0,        9.8         -1.3    1007.7      1638288000.0
2021-12-01  01:00:00    4       0.0                    0.0      0.0,        9.8         -1.3    1007.7      1638288000.0
..........  ........    .       ...                    ....     ...         ...         ....    ...         ......
..........  ........    .       ...                    ....     ...         ...         ....    ...         ......
2021-12-10  17:00:00    2       0.06735057830810548    0.087    23.9        2.3         -1.2    1007.6      163828800.0
2021-12-10  17:00:00    3       0.03403729248046875    0.091    24.1        2.3         -1.2    1007.6      163828800.0
2021-12-10  17:00:00    4       0.08401119232177734    0.09     24.3        2.3         -1.2    1007.6      163828800.0
2021-12-10  17:00:00    5       0.08356260681152344    0.087    24.6        2.3         -1.2    1007.6      163828800.0

Dataset shape after I make train and test set:

(1170, 9)
Training shape: (930, 30, 8)
Testing shape: (185, 30, 8)

This is my code:

import os
import pandas as pd
import numpy as np
from sklearn.preprocessing import MinMaxScaler
#from sklearn.externals import joblib
import joblib
import seaborn as sns
sns.set(color_codes=True)
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from numpy.random import seed
#from tensorflow import set_random_seed
import tensorflow
tensorflow.random.set_seed

import tensorflow as tf
#tf.logging.set_verbosity(tf.logging.ERROR)

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Input, Dropout, Dense, LSTM, TimeDistributed, RepeatVector
from tensorflow.keras.models import Model
from tensorflow.keras import regularizers
import plotly.graph_objects as go

dataset = pd.read_csv('./data/combined.csv')
print(dataset.shape)
dataset.fillna(0, inplace=True)
dataset = dataset.set_index('datetime')

train = dataset[:'2021-12-08 23:00:00']
test = dataset['2021-12-08 23:00:00':]

scaler = StandardScaler()
scaler = scaler.fit(train)

train = scaler.transform(train)
test = scaler.transform(test)

TIME_STEPS=30

def create_sequences(X, y, time_steps=TIME_STEPS):
    Xs, ys = [], []
    for i in range(len(X)-time_steps):
        Xs.append(X.iloc[i:(i+time_steps)].values)
        ys.append(y.iloc[i+time_steps])
    
    return np.array(Xs), np.array(ys)

X_train, y_train = create_sequences(train, train)
X_test, y_test = create_sequences(test, test)

print(f'Training shape: {X_train.shape}')
print(f'Testing shape: {X_test.shape}')

model = Sequential()
model.add(LSTM(128, input_shape=(X_train.shape[1], X_train.shape[2])))
model.add(Dropout(rate=0.2))
model.add(RepeatVector(X_train.shape[1]))
model.add(LSTM(128, return_sequences=True))
model.add(Dropout(rate=0.2))
model.add(TimeDistributed(Dense(X_train.shape[2])))
model.compile(optimizer='adam', loss='mae')
model.summary()

history = model.fit(X_train, y_train, epochs=100, batch_size=16, validation_split=0.1 , shuffle=False)

Whenever I run this code I get following error:

Traceback (most recent call last):
  File "/Users/sudip/Desktop/workspace/local_work/LSTM_api/test-1.py", line 58, in <module>
    X_train, y_train = create_sequences(train, train)
  File "/Users/sudip/Desktop/workspace/local_work/LSTM_api/test-1.py", line 53, in create_sequences
    Xs.append(X.iloc[i:(i+time_steps)].values)
AttributeError: 'numpy.ndarray' object has no attribute 'iloc'

After removing iloc and values I get following error:

Epoch 1/100
Traceback (most recent call last):
  File "/Users/sudip/Desktop/workspace/local_work/LSTM_api/test-1.py", line 77, in <module>
    history = model.fit(X_train, y_train, epochs=100, batch_size=16, validation_split=0.1 , shuffle=False)
  File "/Users/sudip/Desktop/workspace/env/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/Users/sudip/Desktop/workspace/env/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 58, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError:  Incompatible shapes: [16,30,8] vs. [16,8]
         [[node gradient_tape/mean_absolute_error/sub/BroadcastGradientArgs
 (defined at /Users/sudip/Desktop/workspace/env/lib/python3.9/site-packages/keras/optimizer_v2/optimizer_v2.py:464)
]] [Op:__inference_train_function_5593]

Errors may have originated from an input operation.
Input Source operations connected to node gradient_tape/mean_absolute_error/sub/BroadcastGradientArgs:

I think errors are from input shapes. Can I get some help to fix this issue?

How can I split train and test from timeseries data based on date and time?

Solution

There is a problem with the data shape. The input shape and the output shape of your network are the same, but the shapes of X_train and y_train are not.
A simple model that would do the job:

model = Sequential()
model.add(LSTM(128, input_shape=(X_train.shape[1], X_train.shape[2])))
model.add(Dense(y_train.shape[1]))
model.compile(optimizer='adam', loss='mae')
model.summary()

Answered By - elbe