[FIXED] Why is the output of torch.lstsq drastically different than np.linalg.lstsq?

Issue

Pytorch provides a lstsq function, but the result it returns drastically differs from the numpy's version. Here is an example input and both of their results:

import numpy as np
import torch 

a = torch.tensor([[1., 1, 1],
                  [2, 3, 4],
                  [3, 5, 2],
                  [4, 2, 5],
                  [5, 4, 3]])

b = torch.tensor([[-10., -3],
                  [ 12, 14],
                  [ 14, 12],
                  [ 16, 16],
                  [ 18, 16]])

a1 = a.clone().numpy()
b1 = b.clone().numpy()

x, r = torch.lstsq(b, a)

x1, res, r1, s = np.linalg.lstsq(b1, a1)

print(f'torch_x: {x}')
print(f'torch_r: {r}\n')

print(f'np_x: {x1}')
print(f'np_res: {res}')
print(f'np_r1(rank): {r1}')
print(f'np_s: {s}')

Output:

torch_x: tensor([[ 2.0000,  1.0000],
        [ 1.0000,  1.0000],
        [ 1.0000,  2.0000],
        [10.9635,  4.8501],
        [ 8.9332,  5.2418]])
torch_r: tensor([[-7.4162, -6.7420, -6.7420],
        [ 0.2376, -3.0896,  0.1471],
        [ 0.3565,  0.5272,  3.0861],
        [ 0.4753, -0.3952, -0.4312],
        [ 0.5941, -0.1411,  0.2681]])

np_x: [[-0.11452514 -0.10474861 -0.28631285]
 [ 0.35913807  0.33719075  0.54070234]]
np_res: [ 5.4269753 10.197526   1.4185953]
np_r1(rank): 2
np_s: [43.057705  5.199417]

What am I missing here?

Solution

torch.lstq(a, b) solves minX L2∥bX−a∥ while np.linalg.lstsq(a, b) solves minX L2∥aX−b∥

So change the order of parameters passed.

Here's a sample:

import numpy as np import torch

a = torch.tensor([[1., 1, 1],
                  [2, 3, 4],
                  [3, 5, 2],
                  [4, 2, 5],
                  [5, 4, 3]])

b = torch.tensor([[-10., -3],
                  [ 12, 14],
                  [ 14, 12],
                  [ 16, 16],
                  [ 18, 16]])

a1 = a.clone().numpy()
b1 = b.clone().numpy()

x, _ = torch.lstsq(a, b)

x1, res, r1, s = np.linalg.lstsq(b1, a1)

print(f'torch_x: {x[:b.shape[1]]}')

print(f'np_x: {x1}')

Results:

torch_x: tensor([[-0.1145, -0.1047, -0.2863],
        [ 0.3591,  0.3372,  0.5407]])
np_x: [[-0.11452514 -0.10474861 -0.28631285]
 [ 0.35913807  0.33719075  0.54070234]]

link to torch doc link to numpy doc

And also the returned rank from numpy.lianalg.lstsq is rank of 1st parameters . To get rank in pytorch use torch.matrix_rank() function.

Answered By - Girish Dattatray Hegde