[FIXED] How to apply cross_val_score to cross valid our own model

Issue

Usually, we apply cross_val_score to the Sklearn models by doing the following way.

scores = cross_val_score(clf, X, y, cv=5, scoring='f1_macro')

Now I have my own models that I wish to perform cross validation. How should I approach it?

tf.keras.backend.clear_session()


model = tf.keras.models.Sequential()
model.add(Masking(mask_value=0.0, input_shape=(X_train.shape[1], X_train.shape[2])))
model.add(Bidirectional(LSTM(128, dropout=dropout, recurrent_dropout=Rdropout, return_sequences=True)))
# model.add(Bidirectional(LSTM(64, dropout=dropout, recurrent_dropout=Rdropout, return_sequences=True)))
# model.add(Bidirectional(LSTM(128, dropout=dropout, recurrent_dropout=Rdropout, return_sequences=True)))
model.add(Bidirectional(LSTM(32, dropout=dropout, recurrent_dropout=Rdropout)))
# model.add(Dense(6, activation='relu'))
# model.add(Dense(4, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))

adamopt = tf.keras.optimizers.Adam(lr=0.003, beta_1=0.9, beta_2=0.999, epsilon=1e-8)
RMSopt = tf.keras.optimizers.RMSprop(lr=0.0007,rho=0.9, epsilon=1e-6)

model.compile(loss='binary_crossentropy',
              optimizer=RMSopt,
              metrics=['accuracy'])

print(cross_val_score(model, X_train, y_train, cv=2,scoring='accuracy'))

TypeError: Cannot clone object '<tensorflow.python.keras.engine.sequential.Sequential object at 0x7f86481170f0>' (type <class 'tensorflow.python.keras.engine.sequential.Sequential'>): it does not seem to be a scikit-learn estimator as it does not implement a 'get_params' methods.

I think that cross_val_score is exclusive to Sklearn models?

Solution

cross_val_score is indeed exclusive to Sklearn models, or models that implements the same required functions, which is not the case for a Keras model.

There is no pre-build function for Keras that allow you to cross validate your model, you will need to code your cross validation algorithm.

First you should decide how many folds do you want to have, then you can use the KFold class from sklearn to divide your dataset in that many folds. (note that KFold.split returns the indices of the datapoints and not the actual datapoints)

Then, you should train a new model for each split and computes the metrics you want. You can follow this tutorial for more information.

Answered By - robinood