Issue
During training I want to keep the weight of the input layer same, while allowing the bias to be updated. I am using pytorch right now, but want the same for tensor flow too.
Currently as a work around I reset the weight every epoch. This is not efficient. Will like to get access to the internals to disable the update. I am assuming zero_grad_ will not work as it will disable both weight and bias. Thanks
Solution
You can set the requires_grad
flag to False
. For example
import torch
import torch.nn as nn
import torch.optim as optim
layer = nn.Linear(8, 8, bias=True)
layer.weight.requires_grad_(False) # set grad to False for weight but not bias
# grab old weights
old_weight = torch.clone(layer.weight.data)
old_bias = torch.clone(layer.bias.data)
opt = optim.SGD(layer.parameters(), lr=1e-3)
x = torch.randn(1, 8)
y = layer(x)
loss = y.sum()
loss.backward()
opt.step() # update step
new_weight = layer.weight.data
new_bias = layer.bias.data
(new_bias == old_bias).all()
>False
(new_weight == old_weight).all()
>True
You also seem confused about the zero_grad
function. This is for removing old gradients before computing new gradients. It is unrelated to choosing which parameters are updated. You can read more here
Answered By - Karl
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.