Issue
I am having two pretrained models in pytorch which use different type of input data. Both of them I am using for feature extraction and want to concatenate their outputs at the end and put them into LSTM.
I am having the following outputs:
ModelA [batch size, num of features, num of frames]
ModelB [batch size, num of features, num of frames, height, width]
How can I concatenate them so that I can put concatenated vector into LSTM and train it. I am not sure how to do it in a way so that my concatenated vector still "makes sense" and does not lose any information.
Solution
You can reshape the second model's output vector such that the new number of features would be num of features x height x width
Assuming that both vectors have the same batch size and the same number of frames:
First, you reorder both output vectors such that the order is [batch size, num of frames, num of features, ...]
Then, you concatenate on the last dimension:
output1 = torch.randn(10, 5, 20)
output2 = torch.randn(10, 7, 20, 100, 200)
output1 = output1.permute(0, 2, 1)
output2 = output2.permute(0, 2, 1, 3, 4)
output = torch.concatenate([output1, output2], dim=2)
Now, the output tensor has all features from both output vectors together, per frame number per instance in batch which would generally make sense
Answered By - Yakov Dan
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.