[FIXED] value error in cnn- keras fit_generator call back

Issue

Error in model.fit_generator(). Being a beginner level programmer in python, I am not sure what this error indicate.

I am trying transfer learning in python to train VGG19 with imagenet, I encounter a value error in callback. Can anyone suggest the changes I should do on this code??

I tried to execute the following code in colab, but getting errors

!apt-get install -y -qq software-properties-common python-software-properties module-init-tools
!add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null
!apt-get update -qq 2>&1 > /dev/null
!apt-get -y install -qq google-drive-ocamlfuse fuse
from google.colab import auth
auth.authenticate_user()
from oauth2client.client import GoogleCredentials
creds = GoogleCredentials.get_application_default()
import getpass
!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL
vcode = getpass.getpass()
!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}
!mkdir -p drive
!google-drive-ocamlfuse drive
!pip install opencv-python
!pip install opencv-contrib-python
!apt update && apt install -y libsm6 libxext6
!pip install -q keras
from glob import glob

import cv2
import numpy as np

from sklearn.utils import shuffle
from sklearn.model_selection import train_test_split

from keras.utils import to_categorical
from keras.models import Model
from keras.layers import Dropout, Dense, Flatten
from keras.optimizers import SGD
from keras.losses import categorical_crossentropy
from keras.regularizers import l2
from keras.applications.vgg19 import VGG19
model = VGG19(include_top=False, weights='imagenet', pooling='avg')
for layer in model.layers:
    layer.trainable = False
x = model.output
predictions = Dense(7, activation='softmax')(x)
model_final = Model(input=model.input, output=predictions)
from keras.callbacks import ReduceLROnPlateau
lr_reducer = ReduceLROnPlateau(monitor='val_loss', factor=0.9, patience=4, verbose=1)
model_final.compile(loss=categorical_crossentropy,
                  optimizer=SGD(lr=0.001, momentum=0.9, nesterov=True),
                  metrics=['accuracy'])
model_final.fit(np.array(X_train), np.array(y_train),
              batch_size=32,
              epochs=10,
              verbose=1,
              validation_split=0.1,
              shuffle=True)
for layer in model_final.layers[7:]:
  layer.trainable = True
model_final.compile(loss=categorical_crossentropy,
                  optimizer=SGD(lr=0.001, momentum=0.9, nesterov=True),
                  metrics=['accuracy'])
from keras.preprocessing.image import ImageDataGenerator
train_generator = ImageDataGenerator(
    featurewise_center = True,
    featurewise_std_normalization = True,
    rotation_range=30,
    shear_range=0.2,
    zoom_range=0.2,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True)

train_generator.fit(np.array(X_train))

test_generator = ImageDataGenerator(
    featurewise_center = True,
    featurewise_std_normalization = True)

test_generator.fit(np.array(X_train))

model_final.fit_generator(train_generator.flow(np.array(X_train), np.array(y_train), batch_size=32),
                          validation_data=test_generator.flow(np.array(X_test), np.array(y_test)),
                          steps_per_epoch=len(X_train)/32, 
                          epochs=50)

ValueError                                Traceback (most recent call last)
<ipython-input-39-f9af6d0d8994> in <module>()
      2                           validation_data=test_generator.flow(np.array(X_test), np.array(y_test)),
      3                           steps_per_epoch=len(X_train)/32,
----> 4                           epochs=50)

2 frames
/usr/local/lib/python3.6/dist-packages/keras/engine/training_generator.py in fit_generator(model, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
     66     if (val_gen and not isinstance(validation_data, Sequence) and
     67             not validation_steps):
---> 68         raise ValueError('`validation_steps=None` is only valid for a'
     69                          ' generator based on the `keras.utils.Sequence`'
     70                          ' class. Please specify `validation_steps` or use'

ValueError: `validation_steps=None` is only valid for a generator based on the `keras.utils.Sequence` class. Please specify `validation_steps` or use the `keras.utils.Sequence` class.

Solution

You have to specify validation_steps in model_final.fit_generator. It is because generators do not know the total number of data will be used, it only know the batch_size (batch_size=32 by default). So you have to manually tell the generator when to stop loading data by providing the number of step in each epoch. step actually means number of batches.

If you want to use all test data for validation in every epoch:

model_final.fit_generator(train_generator.flow(np.array(X_train), np.array(y_train), batch_size=32),
                          validation_data=test_generator.flow(np.array(X_test), np.array(y_test), batch_size=32),
                          steps_per_epoch=len(X_train)/32, 
                          validation_steps=len(X_test)/32,
                          epochs=50)

The error message also mentioned that validation_step=None is only valid when you use generator inherited from Sequence. In such case, validation_step will be set to len(validation_data) automatically. See here. It can be done only because __len__(self) method is defined in Sequence object See here, but not in your ImageDataGenerator

Answered By - meowongac