[FIXED] How to apply regularization when fine tuning on the imagenet extracted models

Issue

One way of fine tuning is extracting a model ( like VGG16 trained on Imagenet) , adding a layer or so and then train the model.

Is it possible to apply regularization to the model layers apart from the added layer using Tensorflow.Keras. I don't think adding regularization to only one layer effects the outcome much.

I know we can apply the regularization for the added layer as:

x = Dense(classes, kernel_regularizer=l2(reg), name="labels")(x)

But is it possible to apply regularization for other layers as well in Keras. It could be easily done in mxnet.

Would be grateful for any help.

Solution

This solution should work. By iterating over the model layers we can just add a regularizer. You can then add your dense layer after this.

model = tf.keras.applications.VGG16(include_top=False, weights=None)
regularizer = tf.keras.regularizers.l2(0.001)

for i in range(len(model.layers)):
    if isinstance(model.layers[i], tf.keras.layers.Conv2D):
        print('Adding regularizer to layer {}'.format(model.layers[i].name))
        model.layers[i].kernel_regularizer = regularizer

# Add Dense layer
classes = 10
x = model.output
x = Dense(classes, kernel_regularizer=regularizer, name="labels")(x)

model = tf.keras.Model(model.input, x)

Answered By - DMolony