[FIXED] correlating between prediction result to label

Issue

I've a keras model that predicted the following results: (it's a multi-class problem with 6 possible classes)

[[0.44599777 0.00667355 0.10674711 0.02558559 0.29180232 0.12319366]]

so given the above results the model predicts the 1st class - but I know it's wrong.

I was able to achieve a ~92% accuracy:

Epoch 1/10
1128/1128 [==============================] - 18s 15ms/step - loss: 1.3685 - accuracy: 0.4596 - val_loss: 0.6238 - val_accuracy: 0.7785
Epoch 2/10
1128/1128 [==============================] - 17s 15ms/step - loss: 0.7200 - accuracy: 0.7373 - val_loss: 0.4055 - val_accuracy: 0.8467
Epoch 3/10
1128/1128 [==============================] - 17s 15ms/step - loss: 0.4994 - accuracy: 0.8200 - val_loss: 0.3284 - val_accuracy: 0.8772
Epoch 4/10
1128/1128 [==============================] - 17s 15ms/step - loss: 0.3966 - accuracy: 0.8568 - val_loss: 0.3100 - val_accuracy: 0.9043
Epoch 5/10
1128/1128 [==============================] - 18s 16ms/step - loss: 0.3428 - accuracy: 0.8810 - val_loss: 0.3044 - val_accuracy: 0.9102
Epoch 6/10
1128/1128 [==============================] - 39s 34ms/step - loss: 0.3075 - accuracy: 0.8915 - val_loss: 0.2970 - val_accuracy: 0.9184
Epoch 7/10
1128/1128 [==============================] - 25s 22ms/step - loss: 0.2779 - accuracy: 0.9035 - val_loss: 0.3051 - val_accuracy: 0.9226
Epoch 8/10
1128/1128 [==============================] - 19s 17ms/step - loss: 0.2663 - accuracy: 0.9069 - val_loss: 0.3207 - val_accuracy: 0.9261
Epoch 9/10
1128/1128 [==============================] - 19s 17ms/step - loss: 0.2514 - accuracy: 0.9138 - val_loss: 0.2855 - val_accuracy: 0.9311
Epoch 10/10
1128/1128 [==============================] - 20s 18ms/step - loss: 0.2331 - accuracy: 0.9196 - val_loss: 0.3352 - val_accuracy: 0.9263
Test loss: 0.33516398072242737
Test accuracy: 0.9262799024581909

Below is how I'm doing the prediction:

bug_name = '51859'
issue = conn.issue(bug_name, expand='changelog')
candidate_bug = Bug(issue, connections_dict)
candidate_bug.extract_all_info()
data = candidate_bug.get_data_as_df()
data = data.drop('group_name', axis='columns')

free_text_tokenized, _ = prepare_free_text_inputs(data, data)

model_inputs = [free_text_tokenized]

res = model.predict(model_inputs)
print(f'expected: {get_group_by_bug_owner(candidate_bug.get_owner())}')
# Generate arg maxes for predictions
print(res)
classes = np.argmax(res, axis=1)
print(classes)
print(np.unique(y_train))
class_index = classes[0]
print(np.unique(y_train)[class_index])

and here's the output:

expected: D
[[0.44599777 0.00667355 0.10674711 0.02558559 0.29180232 0.12319366]]
[0]
['A' 'B' 'C' 'D' 'E' 'F']
A

... so I'm afraid my problem is I don't know to "assign" those results to the labels. I've tried multiple attempts (where I know what the prediction should be) and it always misses the expected result.

Also - I'm using LabelEncoder as follows:

    
# prepare target
def prepare_targets(y_train, y_test):
    le = LabelEncoder()
    le.fit(y_train)
    y_train_enc = le.transform(y_train)
    y_test_enc = le.transform(y_test)
    return y_train_enc, y_test_enc

y_train_enc, y_test_enc = prepare_targets(y_train, y_test)

What am I missing? Am I using the wrong the wrong list (y_train)?

Solution

Answering my own question (for whoever's gonna be introduced by it).

2 Issues I've found:

1. I've (very mistakenly) triggered the transformers on the predicted data (fit_on_text) and it's a big no no! - one's must use the same transformer that was already fitted via the trained data.

2. the labels are encoded in the LabelEncoder that was originally used before training the model, so I've created a dict to map each label as follows:

# prepare target
print('preparing lables')
le = LabelEncoder()
le_name_mapping = {}

le.fit(y_train)
le_name_mapping.update(dict(zip(le.transform(le.classes_), le.classes_)))
print(le_name_mapping)
y_train_enc = le.transform(y_train)
y_test_enc = le.transform(y_test)

later on I've used it on the prediction results:

res = model.predict(model_inputs)
selected_class_index = np.argmax(res, axis=1)[0]
print(selected_class_index)
print(f'actual: {le_name_mapping[selected_class_index]}')

Answered By - Ben