[FIXED] How can I add some error to linear layer in pytorch?

Issue

I want to add some errors to parameters like the following code.

import torch
import torch.nn as nn

layer = nn.Linear(10, 100)
plus_weight = torch.randn(100, 10)
layer.weight += plus_weight

But I got this error.

RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

How can I avoid this error? And why can't I change the layer parameter value directly?

Solution

you can use .data to modify the value of leaf node

layer.weight.data += plus_weight

if you want initialize or update parameters, assigning to .data is the way to go. But it is very dangerous to take a in-place operation to modify the value of a leaf node in the middle of a forward pass since it may broken the backward result.

For example

import torch
a = torch.randn(1,1,requires_grad=True)
print(a)
b = a**2
c = torch.sum(b**2)
a.data += 1
c.backward()
print(a.grad)

the output result will be

tensor([[0.9123]], requires_grad=True)
tensor([[6.3666]])

with the code above, we have c = b**2 = a**2。 Derivation of c with respect to a should should be 4*a**3. but since you update the value of a during middle of forward process, the b will be 0.9123**2 not (0.9123+1)**2. and the final result with a.grad will be 2*(0.9123**2)*2*1.9123

Answered By - pythonHua