[FIXED] Convert batch normalization from Tensorlayer tf1.x to TF2 keras

Issue

Trying to convert the batch normalization layer from Tensorlayer version 1.11.1 to Tensorflow 2 and getting different outputs from this layer during inference using the same pretrained model.

Tensorlayer 1.11.1

tensorlayer.layers.BatchNormLayer(network, is_train=False, name="batch_norm")

Tensorflow 2.8.0

tf.keras.layers.BatchNormalization(trainable=False, momentum=0.9, axis=3, epsilon=1e-05, gamma_initializer=tf.random_normal_initializer(mean=1.0, stdev=0.002))(network)

What am I missing to get the BatchNorm output to match?

Solution

The TF1 model I had was in NPZ format. The weights from Tensorlayer are saved in the order of: beta, gamma, moving mean, variance. In TF2, the batch norm layer is in the order of: gamma, beta, moving mean, variance.

If the order of the weights for beta and gamma are reversed when moving from TF1 to TF2 it solves the issue.

Answered By - ro5423