[FIXED] What is the formal definition for "view" in this context?

Issue

Consider the following paragraph from the sub-section named The essence of tensors from the section named Tensors: Multidimensional arrays of the chapter named It starts with a tensor from the book titled Deep Learning with PyTorch by Eli Stevens et al.

Python lists or tuples of numbers are collections of Python objects that are individually allocated in memory, as shown on the left in figure 3.3. PyTorch tensors or NumPy arrays, on the other hand, are views over (typically) contiguous memory blocks containing unboxed C numeric types rather than Python objects. Each element is a 32-bit (4-byte) float in this case, as we can see on the right side of figure 3.3. This means storing a 1D tensor of 1,000,000 float numbers will require exactly 4,000,000 contiguous bytes, plus a small overhead for the metadata (such as dimensions and numeric type).

And the figure they are referring to is shown below, taken from the book

The above paragraph is saying that tensors are views over contiguous memory blocks. What exactly is meant by a view in this context?

Solution

A "view" is how you interpret this data, or more precisely, the shape of the tensor. For example, given a memory block with 40 contiguous bytes (10 contiguous floats), you can either view it as a 2x5 tensor, or a 5x2 tensor.

In pytorch, the API to change the view of a tensor is view(). Some examples:

Python 3.8.10 (default, Sep 28 2021, 16:10:42) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> x = torch.randn(10, dtype=torch.float32)
>>> x.shape
torch.Size([10])
>>>
>>> x = x.view(2, 5)
>>> x.shape
torch.Size([2, 5])
>>>
>>> x = x.view(5, 2)
>>> x.shape
torch.Size([5, 2])

Of course, some views are forbidden for 10 floats:

>>> x = x.view(3, 3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: shape '[3, 3]' is invalid for input of size 10

view does not change the data in the underlying memory. It merely changes how you "view" the tensor.

Answered By - ihdv