[FIXED] How do I load a non-latest Tensorflow checkpoint?

Issue

I made checkpoints every 1000 steps of training, and I have 16 files in my checkpoints directory. However it seems that when I want to retrieve the latest one it's reverting to its pre-trained state. I am assuming something to do with the summary logs not documenting that later checkpoints exist.

chkpt.restore(tf.train.latest_checkpoint(chkpt_dir))
# fit(train_ds, test_ds, steps=100000)

for i in range(10):
    ex_input, ex_output = next(iter(test_ds.take(1)))
    generate_images(generator, ex_input, ex_output, i, test=True)

How can I manually ask the checkpoint manager to retrieve this or that particular checkpoint file, as oppossed to .latest_checkpoint()?

Edit: Solved it myself, open the checkpoints.txt file in your checkpoint folder and set the suffix number to whichever checkpoint you want to load.

Solution

you can use the checkpoints.restore() method to restore checkpoints of your preference. For example, if you want to load checkpoint at iteration 1000, then you write:

checkpoint.restore('./test/model.ckpt-1000')

For more details please refer to this documentation. Thank You.

Answered By - TFer