[FIXED] How are tensors of different shapes combined?

Issue

In what logic are multidimensional tensors (e.g. in pytorch) combined? Let's say I have tensor A with shape (64, 1, 1, 42) and tensor B with shape (1, 42, 42). What is the result of A & B and how can I determine the resulting shape (if possible) in advance?

Solution

The combination of tensors of different shapes is often accomplished by a mechanism called Broadcasting. Broadcasting is a powerful tool in libraries like NumPy and PyTorch, which helps to expand the size of tensors without actually copying the data, thus performing operations between tensors of different shapes. Broadcast rules are generally consistent between NumPy and PyTorch.

Broadcasting rules: If the number of dimensions of the tensors doesn't match, prepend 1 to the dimensions of the tensor with fewer dimensions.

Compare the size of each dimension:

If the size matches or one of the sizes is 1, then broadcasting is possible for that dimension. If neither size is 1 and the sizes don't match, broadcasting fails. After successful broadcasting, each tensor behaves as if its shape were the element-wise maximum of the shapes of the two input tensors.

In any dimension where one tensor has size 1, and the other tensor has a size greater than 1, the smaller tensor behaves as if it had been expanded to match the size of the larger tensor. For example:

Given:

Tensor A of shape (64, 1, 1, 42) Tensor B of shape (1, 42, 42)

To combine them:

Make the number of dimensions equal:

Tensor A: (64, 1, 1, 42) Tensor B: (1, 1, 42, 42) Compare the dimensions:

The dimensions are (64, 1, 1, 42) and (1, 1, 42, 42) Every dimension is either the same or one of them is 1, so broadcasting is possible. The resulting tensor will have the shape: (64, 1, 42, 42)

import torch

# Creating example tensors
A = torch.rand((64, 1, 1, 42))
B = torch.rand((1, 42, 42))

# Broadcasting operation
result = A * B

# Outputting the resulting shape
print(result.shape)  

which results in

torch.Size([64, 1, 42, 42])

as expected

Edit: To address the first comment to this answer:

thanks! so each dimension must either be one or the dimensions of both tensors must match. is rearranging also done by default if the previously mentioned check fails but a rearranging would help?

Correct, for two tensors to be combined using broadcasting in PyTorch. Each dimension of both tensors is either 1, or they match. PyTorch starts comparing dimensions from the trailing dimension (the last one) and works its way backward.

If the broadcasting criteria aren't met, you'll get a runtime error.

However, if the broadcasting criteria fail, you might be able to rearrange or reshape the tensors to make them compatible:

Adding Singleton Dimensions: You can use the unsqueeze method or numpy-style None indexing. For instance, if you have a tensor of shape (5, 4) and you want it to have a shape (5, 1, 4), you can add a singleton dimension.

tensor = tensor.unsqueeze(1) 
# or
tensor = tensor[:, None, :]

Removing Singleton Dimensions: You can use the squeeze method to remove singleton dimensions. For example, to change a tensor from shape (5, 1, 4) to (5, 4).

tensor = tensor.squeeze(1)

Reshape: If adding or removing singleton dimensions isn't enough, you might need to fully reshape your tensor using the reshape method.

tensor = tensor.reshape(new_shape)

Permute Dimensions: If the problem is the order of dimensions, you can rearrange them using the permute method.

tensor = tensor.permute(dimension_order)

It's essential to know the semantics of your data when rearranging or reshaping tensors. By changing the shape, you might alter the tensor's meaning or relationship between its values. Always ensure that such operations make sense in the context of your problem.

Answered By - Robert Long