[FIXED] Convert weight and bias to sparse tensor pytorch

Issue

I'm trying to convert torch.nn.Parameters to sparse tensor. Pytorch documents say that Parameters is a Tensor's subclass. Tensor support to_sparse method but if I convert a Parameters to sparse, it will give me:
TypeError: cannot assign 'torch.cuda.sparse.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)
Is there a way to bypass this and use sparse tensor for Parameters?
Here is example code to produce the problem:

for name, module in net.named_modules():
    if isinstance(module, torch.nn.Conv2d):
        module.weight = module.weight.data.to_sparse()
        module.bias = module.bias.data.to_sparse()

Solution

torch.Tensor.to_sparse() returns a sparse copy of the tensor which cannot be assigned to module.weight since this is an instance of torch.nn.Parameter. So, you should rather do:

module.weight = torch.nn.Parameter(module.weight.data.to_sparse())
module.bias = torch.nn.Parameter(module.bias.data.to_sparse())

Please note that Parameters are a specific type of Tensor that is marked as being a parameter from an nn.Module, so they are different from ordinary Tensors.

Answered By - Wasi Ahmad