[FIXED] why Changing in numpy array change tensor that converted from numpy?

Issue

When I convert a numpy array to tensor and change the numpy in differnet ways some change the tensor and some not. it is sample:

in this code:

array = np.arange(1., 8.)
tensor = torch.from_numpy(array)
array, tensor

the answer is :

(array([1., 2., 3., 4., 5., 6., 7.]),
 tensor([1., 2., 3., 4., 5., 6., 7.], dtype=torch.float64))

No when I add 1 to numpy array in this way it change tensor values:

array = np.arange(1., 8.)
tensor = torch.from_numpy(array)
array += 1
array, tensor

The output is:

(array([2., 3., 4., 5., 6., 7., 8.]),
 tensor([2., 3., 4., 5., 6., 7., 8.], dtype=torch.float64))

But when I change numpy array in another way it dos not change tensor:

array = np.arange(1., 8.)
tensor = torch.from_numpy(array)
array = array + 1
array, tensor

the output is :

(array([2., 3., 4., 5., 6., 7., 8.]),
 tensor([1., 2., 3., 4., 5., 6., 7.], dtype=torch.float64))

again this code dos not change it:

array = np.arange(1., 8.)
tensor = torch.from_numpy(array)
array = array + 1
array += 1
array, tensor

This also happened when changing pytorch's tensor to numpy array.

Why this way work numpy and torch?

Solution

TL;DR:

+= modifies in place but x = x + 1 creates a copy.

Long version:

You may want to read about pinned memory. I suspect PyTorch is using it to reduce data transfer overheads for better performance. When you re-assign using array = array + 1 a copy is created (you can try printing the address of the array using id(array) to check if the address is changed after the assignment, meaning a copy is made). When a copy is made, array and tensor no longer point to the same memory address, so modifying array no longer updates the pinned memory.

import numpy as np
arr = np.arange(3)
print((arr, id(arr)))
arr+=1
print((arr, id(arr)))
arr = arr + 1
print((arr, id(arr)))

which (for me) results in:

(array([0, 1, 2]), 1951948494416)
(array([1, 2, 3]), 1951948494416)
(array([2, 3, 4]), 1952052860880)

When you create a copy, the old array is still being used by the GPU so it remains there (memory isn't freed) so you can still access its old content (check pass by reference vs pass by value).

Answered By - OM222O