[FIXED] How to create a Pytorch network with mixed categorical and continuous matrix input

Issue

I'm creating a network network that will take a matrix of continuous values along with some categorical input represented as vectors of all the classes.

Now, I'm also looking extract features from the matrix with convolution. But this would not be possible if I reduce the matrix to dimension 1 and concatenate with the class vectors.

Is there a way to concat it together as a single input? Or do I have to create two separate input layer then somehow join them after the convolution? If its the latter, what function am I looking for?

Solution

The most common approach to create continuous values from categorical data is nn.Embedding. It creates a learnable vector representation of the available classes, such that two similar classes (in a specific context) are closer to each other than two dissimilar classes.

When you have a vector of classes with size [v], the embedding would create a tensor of size [v, embedding_size], where each class is represented by a vector of length embedding_size.

num_classes = 4
embedding_size = 10

embedding = nn.Embedding(num_classes, embedding_size)

class_vector = torch.tensor([1, 0, 3, 3, 2])

embedded_classes = embedding(class_vector)
embedded_classes.size() # => torch.Size([5, 10])

How you combine them with your continuous matrix depends on your particular use case. If you just want a 1D vector you can flatten and concatenate them. On the other hand, if the matrix has meaningful dimensions that you want to keep, you should decide which dimension makes sense to concatenate on and adapt the embedding_size such that they can be concatenated.

Answered By - Michael Jungo