[FIXED] Initial weights in Network with Keras / Tensorflow

Issue

I am trying to get the initial weights for a given network.

This thread suggests that one needs to specify the input dimension: How to view initialized weights (i.e. before training)?

This thread suggests that after compilation weights should be available: Reset weights in Keras layer

Save the initial weights right after compiling the model but before training it.

While I reproduced the results in the first post I am unable to apply it to my case:

import numpy as np
from keras.models import Sequential
from keras.layers import Dense
import tensorflow as tf

# Reproducing https://stackoverflow.com/questions/46798708/how-to-view-initialized-weights-i-e-before-training

# First model without input_dim prints an empty list
model = Sequential()
model.add(Dense(5, weights=[np.ones((3, 5)), np.zeros(5)],
                activation='relu'))
print(model.get_weights())

# Second model with input_dim prints the assigned weights
model1 = Sequential()
model1.add(Dense(5, weights=[np.ones((3, 5)), np.zeros(5)],
                 input_dim=3,
                 activation='relu'))
model1.add(Dense(1, activation='sigmoid'))

print(model1.get_weights())


class Test(tf.keras.Model):
    def __init__(self, n_inputs: int, neurons=10):
        super(Test, self).__init__(name="Test")
        self.neurons = neurons

        # Initilializers
        mean, std = 0., 0.0005
        bias_normalization = None
        kernel_initializer = tf.keras.initializers.RandomNormal(mean=mean,
                                                                stddev=std)
        self.h1 = Dense(n_inputs, activation="linear", name="h1",
                        kernel_initializer=kernel_initializer,
                        bias_initializer=bias_normalization,
                        input_dim=n_inputs)

    def call(self, inputs):
        x = self.h1(inputs)
        return x


# Model Test

test = Test(n_inputs=1, neurons=100)
test.get_weights()  # empty, expected

test.compile()
test.get_weights()  # empty, unexpected

Solution

In your case, I think it all depends on when the call method of tf.keras.Model is actually called. Also, Keras sequential models and subclassing models behave differently.

Your model's weights are only created when you pass real data or explicitly call build(*). For example, if you try the following you will get an output of some weights:

test_model = Test(n_inputs=1, neurons=100)
test_model(np.random.random((32, 1)))
print(test_model.get_weights())
# [array([[0.00057544]]), array([0.3752869])]

or

test_model.build(input_shape=(32, 1))
print(test_model.get_weights())
# [array([[8.942684e-05]], dtype=float32), array([-1.6799461], dtype=float32)]

Basically, the call method internally calls the __call__ method. If you check out the official Tensorflow website you can read about this behavior:

To call a model on an input, always use the call method, i.e. model(inputs), which relies on the underlying call method.

You could by the way also define your Test class as follows:

class Test(tf.keras.Model):
    def __init__(self, n_inputs: int, neurons=10):
        super(Test, self).__init__(name="Test")
        self.neurons = neurons

        # Initilializers
        mean, std = 0., 0.0005
        bias_normalization = None
        kernel_initializer = tf.keras.initializers.RandomNormal(mean=mean,
                                                           stddev=std)
        model_input = tf.keras.layers.Input(shape=(n_inputs,))
        x= tf.keras.layers.Dense(n_inputs, activation="linear", name="h1",
                        kernel_initializer=kernel_initializer,
                        bias_initializer=bias_normalization)(model_input)
        self.model = tf.keras.Model(model_input, x)
    

test_model = Test(n_inputs=1, neurons=100)
print(test_model.get_weights())
# [array([[0.00045629]], dtype=float32), array([0.9945322], dtype=float32)]

Answered By - AloneTogether