[FIXED] How to feed the input images into ensemble of DNN models?

Issue

I am trying to ensemble 5 transfer learning pre-trained DNN models (the base models trained on imagenet) using the code below

def define_stacked_model(members):
    # update all layers in all models to not be trainable
    for i in range(len(members)):
        model = members[i]
        for layer in model.layers:
            # make not trainable
            layer.trainable = False
            # rename to avoid 'unique layer name' issue
            layer._name = 'ensemble_' + str(i+1) + '_' + layer.name
    # define multi-headed input
    ensemble_visible = [model.input for model in members]
    # concatenate merge output from each model
    ensemble_outputs = [model.output for model in members]
    merge = concatenate(ensemble_outputs)
    hidden = tf.keras.layers.Dense(10, activation='relu')(merge)
    output = tf.keras.layers.Dense(1)(hidden)
    model = tf.keras.Model(inputs=ensemble_visible, outputs=output)
    # plot graph of ensemble
    plot_model(model, show_shapes=True, to_file='model_graph.png')

    # compile
    model.compile(optimizer='adam',
                   loss=tf.keras.losses.BinaryCrossentropy(from_logits=True), 
                   metrics=["accuracy"])
    return model

# define ensemble model
stacked_model = define_stacked_model(members)
stacked_model.summary()

each model accept input image with size 160X160. I read the images using:

IMG_WIDTH=160
IMG_HEIGHT=160
def create_dataset(img_folder):
   
    img_data_array=[]
    class_name=[]
   
    for dir1 in os.listdir(img_folder):
        for file in os.listdir(os.path.join(img_folder, dir1)):
       
            image_path= os.path.join(img_folder, dir1,  file)
            image = cv2.imread( image_path, cv2.IMREAD_COLOR)
            image = cv2.resize(image, (IMG_HEIGHT, IMG_WIDTH),interpolation = cv2.INTER_AREA)
            image = np.array(image)
            image = image.astype('float32')
            image /= 255 
            img_data_array.append(image)
            class_name.append(dir1)
    return img_data_array, class_name
# extract the image array and class name
img_data, class_name = create_dataset(TRAIN_DIR)

target_dict={k: v for v, k in enumerate(np.unique(class_name))}

target_val=  [target_dict[class_name[i]] for i in range(len(class_name))]
target_val = np.array(list(map(int,target_val))[0:300], np.float32)

note: I used 300 images only because of the availability of the memory.

after that I stacked the images using:

stacked_img_data = np.dstack([img_data[0:300],
                    img_data[0:300],
                    img_data[0:300],
                    img_data[0:300],
                    img_data[0:300]])

and finally trained the ensemble model using:

history = stacked_model.fit(x = stacked_img_data, y = target_val, epochs=10)

I faced this error:

ValueError: Layer model expects 5 input(s), but it received 1 input tensors. Inputs received: [<tf.Tensor 'IteratorGetNext:0' shape=(None, 160, 800, 3) dtype=float32>]

I think the above method is close to the correct way. but I am not sure how to solve the issue.

Solution

Well you can make this work only by changing this history = stacked_model.fit(x = stacked_img_data, y = target_val, epochs=10) to this

   history = stacked_model.fit(x = [img_data[0:300],img_data[0:300],img_data[0:300],img_data[0:300],img_data[0:300]], y = target_val, epochs=10)

That being said you can also do this

input = tf.keras.layers.Input(shape=(160,160,3)) 
ensemble_outputs = [model(input) for model in members]
.
.
model = tf.keras.Model(inputs=input, outputs=output)

and only pass the image one time

Answered By - Pedro Fillastre