[FIXED] Is it true that `inplace=True` activations in PyTorch make sense only for inference mode?

Issue

According to the discussions on PyTorch forum :

• What’s the difference between nn.ReLU() and nn.ReLU(inplace=True)?
• Guidelines for when and why one should set inplace = True?

The purpose of inplace=True is to modify the input in place, without allocating memory for additional tensor with the result of this operation.

This allows to be more efficient in memory usage but prohibits the possibility to make a backward pass, at least if the operation decreases the amount of information. And the backpropagation algorithm requires to have intermediate activations saved in order to update the weights.

Can one say, that this mode, should be turned on in layers only if the model is already trained, and one doesn't want to modify it anymore?

Solution

According to your question:

• nn.ReLU(inplace=True) is for in place operator to save memory either during training and testing.

However there are some problems we may face when we use nn.ReLU(iplace=True) while calculating gradients but it is not the problem of ReLU. Let's say your model has ReLU function with inplace=True, in some cases you need to use some operation during forwarding, when there are other inplace operations in your forward function, then you will have problems with updating gradients otherwise there is no any problem with inplace=True:

def forward(self, x):
    skip = x
    x = self.relu(x)
    x += skip # inplace addition

Because you can not use two inplace operators as shown above. However when you use first addition then activation function with inplace=True, it won't be a problem:

def forward(self, x):
    skip = x
    x += skip # inplace addition
    x = self.relu(x)
    

Answered By - yakhyo