[FIXED] How to reset all variable in Adam Optimizer (Tensorflow), then I can retrain model with new optimizer

Issue

I am doing something related to curriculum learning, therefore I want to reset my optimizer for each training stage. However in Adam, there are some slots maintaining the first moment and second moment, also the power value of beta1 and beta2. I have try to define the tensorflow saver without watching those Variable as below:

def define_saver(exclude='(r'.*Adam.*', r'.*beta.*')'):
"""Create a saver for the variables we want to checkpoint.

Args:
  exclude: List of regexes to match variable names to exclude.

Returns:
  Saver object.
"""
variables = []
exclude = exclude or []
exclude = [re.compile(regex) for regex in exclude]
for variable in tf.global_variables():
    if any(regex.match(variable.name) for regex in exclude):
        continue
    variables.append(variable)
saver = tf.train.Saver(variables, keep_checkpoint_every_n_hours=5)
return saver

I am not sure am I doing right or not, because I have no much idea how Adam work in details, hope someone can share his/her ideas. thanks.

Solution

You can access the state of an Optimizer through its variables member function. If you group all their initializer, you end up with a reset op for the optimizer.

opt = tf.train.AdamOptimizer()
...
opt_reset = tf.group([v.initializer for v in opt.variables()])

Answered By - P-Gn