[FIXED] PyTorch TransformerEncoderLayer different input order gets different results

Issue

Before I start, I’m very new to Transformers, and sorry for by bad sentence structure, I have a fever right now.

Any time I use nn.TransformerEncoderLayer in anyway with a saved model if the data is in a different order I get different results. Is there a way to save the Encode table (or whatever this would be), this would be in the MultiheadAttention part of the TransformerEncoderLayer right?

Just using TransformerEncoderLayer and save the model and then use np.random.permutation() to shuffle the input data. And running the input data through the model after calling model.eval. This always gives me different results unless I use the same order every time.

I have this layer in my model like this

self.transformer = nn.TransformerEncoderLayer()

I save the model like so

torch.save(model, path)

Does this not save the nn.TransformerEncoderLayer() or something?

Solution

Got it, what is the shape of your input, and are you using batch_first=True or not? Basically one thing to just make sure of is that you don’t have your batch and the sequence dimensions mixed up in your implementation.

batch_first – If True, then the input and output tensors are provided as (batch, seq, feature). Default: False (seq, batch, feature).

Answered By - Samuel Fipps