[FIXED] How can I simulate this paragraph about changing the learning rate picked from Resnet paper?

Issue

While going through this paper, I found they are changing learning rate in between of training/validation iterations.

We start with a learning rate of 0.1, divide it by 10 at 32k and 48k iterations, and terminate training at 64k iterations, which is determined on a 45k/5k train/val split.

Solution

Generally, You're probably looking for torch.optim.lr_scheduler

Specifically, You can implement the reduction of learning_rate after n epochs using lr_scheduler.MultiStepLR

Following the advised way to use lr_schedulers, You will need to recalculate the milestones from steps to epochs as the updates are done after the whole epochs rather than steps.

If that won't give You the satisfactory result, You can actually "cheat" by calling scheduler.step() after each batch (step) and then passing the milestones in number of steps.
Remember then not to confuse yourself or the ones that will happen to edit Your code some day, at least put a comment to indicate that You're using the library function in a little less obvious way :D

Answered By - philvec