[FIXED] Keras Categorical_crossentropy loss implementation

Issue

I'm trying to reimplement the Categorical Cross Entropy loss from Keras so that I can customize it. I got the following

def CustomCrossEntropy(output, target, axis=-1):
    target = ops.convert_to_tensor_v2_with_dispatch(target)
    output = ops.convert_to_tensor_v2_with_dispatch(output)
    target.shape.assert_is_compatible_with(output.shape)
    # scale preds so that the class probas of each sample sum to 1
    output = output / math_ops.reduce_sum(output, axis, True)
    # Compute cross entropy from probabilities.
    epsilon_ = _constant_to_tensor(epsilon(), output.dtype.base_dtype)
    output = clip_ops.clip_by_value(output, epsilon_, 1. - epsilon_)
    return -math_ops.reduce_sum(target * math_ops.log(output), axis)

It produces different results than the internal function which confuses me, as I just copied the code from github so far. What am I missing here?

Prove:

y_true = [[0., 1., 0.], [0., 0., 1.]]
y_pred = [[0.05, 0.95, 0], [0.1, 0.8, 0.1]]
loss = tf.keras.losses.categorical_crossentropy(y_true, y_pred)
customLoss = CustomCrossEntropy(y_true, y_pred)
assert loss.shape == (2,)
print(loss)
print(customLoss)
>>tf.Tensor([0.05129331 2.3025851 ], shape=(2,), dtype=float32)
>>tf.Tensor([ 0.8059049 14.506287 ], shape=(2,), dtype=float32)

Solution

You have inverted the arguments of the function in your definition of CustomCrossEntropy, if you double check the source code in GitHub you will see that the first argument is target and the second one is output. If you switch them back you will get the same results as expected.

import tensorflow as tf
from tensorflow.keras.backend import categorical_crossentropy as CustomCrossEntropy

y_true = [[0., 1., 0.], [0., 0., 1.]]
y_pred = [[0.05, 0.95, 0], [0.1, 0.8, 0.1]]

y_true = tf.convert_to_tensor(y_true)
y_pred = tf.convert_to_tensor(y_pred)

loss = tf.keras.losses.categorical_crossentropy(y_true, y_pred)
print(loss)
# tf.Tensor([0.05129331 2.3025851 ], shape=(2,), dtype=float32)

loss = CustomCrossEntropy(y_true, y_pred)
print(loss)
# tf.Tensor([0.05129331 2.3025851 ], shape=(2,), dtype=float32)

loss = CustomCrossEntropy(y_pred, y_true)
print(loss)
# tf.Tensor([ 0.8059049 14.506287 ], shape=(2,), dtype=float32)

Answered By - Flavia Giammarino