[FIXED] Custom loss function in keras? Correntropy. Math/implementation issues

Issue

Im working with a paper where it implements an autoencoder with a custom loss function to work with vibration signals.

Im having trouble implementing it on keras. They implement "Maximum Correntropy" as the loss function to avoid issues with background noise on a signal.

This is the description:

Gaussian kernel is the most popular Mercer kernel in correntropy, which is defined as

where r is the kernel size. Then, the new autoencoder loss function can be designed by maximizing the following function:

Since i never implemented a custom loss function im having issues with the math in python. The kernel is used on the loss function that i need to implement. This is what i have:

dataset.npz

file = np.load('./data/CWRU_48k_load_1_CNN_data.npz')  # Numpy Array 

data = file['data'].reshape(len(file['data']), 1024)
labels = file['labels']
category_labels = np.unique(labels)
labels = pd.Categorical(labels, categories = category_labels).codes

train_data, test_data, train_labels, test_labels = train_test_split(data, labels, test_size = int(data.shape[0]*0.2), random_state = 100, stratify = labels)

train_labels = to_categorical(train_labels)
test_labels = to_categorical(test_labels)

# Data shape. Sample Len: 1024. Outputs/Classifications: 10
print(train_data.shape, train_labels.shape, test_data.shape, test_labels.shape)
#(3680, 1024) (3680, 10) (920, 1024) (920, 10)

act_func = 'relu'
out_func = 'softmax'
k_inic = 'glorot_uniform'  

def create_model(shape=[512, 100], loss_func='mse'):
    model = Sequential()

    for shape_size in shape:
        model.add(Dense(shape_size, activation=act_func, kernel_initializer=k_inic))

    model.add(Dense(10, activation=out_func, kernel_initializer=k_inic))
    model.compile(loss=loss_func, optimizer=keras.optimizers.Adam(), metrics=["accuracy"])
    model.build(input_shape=(None, 1024))

    return model

BATCH_SIZE = 45
EPOCHS = 200
VALIDATION_SPLIT = 0.05

# Design Mercer Kernel
def kernel(x, sigma=1):
    return (1/(K.sqrt(2*np.pi)*sigma))*K.exp((-(x*x)/(2*sigma*sigma)))

# Use Mercer Kernel on Maximum Correntropy for loss function
def correntropy(y_true, y_pred):
    sum_score = 0.0
    for i in range(len(y_true)):
        sum_score = kernel(y_true[i] - y_pred[i])
    sum_score = sum_score/len(y_true)
    return -sum_score

# Create AutoEncoder model with my custom loss function
model = create_model(shape=[512, 100], loss_func=correntropy)
history = model.fit(train_data, train_labels, epochs = EPOCHS, batch_size = BATCH_SIZE, validation_data=(test_data, test_labels), 
                        callbacks = callbacks.callbacks, verbose = 0)

res = model.evaluate(test_data, test_labels, batch_size = BATCH_SIZE, verbose = 0)[1]

But i have this error:

AttributeError: in user code:

    /home/user/.local/lib/python3.8/site-packages/keras/engine/training.py:853 train_function  *
        return step_function(self, iterator)
    /tmp/ipykernel_95935/2003563015.py:26 correntropy  *
        sum_score = kernel(y_true[i] - y_pred[i])
    /tmp/ipykernel_95935/2239884018.py:20 kernel  *
        return (1/(K.sqrt(2*np.pi)*sigma))*K.exp((-(x*x)/(2*sigma*sigma)))
    /home/user/.local/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py:206 wrapper  **
        return target(*args, **kwargs)
    /home/user/.local/lib/python3.8/site-packages/keras/backend.py:2539 sqrt
        zero = _constant_to_tensor(0., x.dtype.base_dtype)

    AttributeError: 'float' object has no attribute 'dtype'

The error seems to be on the kernel, but how do i fix to work with tensors?

print(y_true)
print(y_pred)
>> Tensor("IteratorGetNext:1", shape=(None, 10), dtype=float32)
>> Tensor("sequential_161/dense_491/Softmax:0", shape=(None, 10), dtype=float32)

Solution

There's 3 main things I notice in your code:

1. You are combining math functions from different packages (K, np). Stick to to native tensorflow functions as much as possible (e.g. tf.math.reduce_sum). There's lots of stuff. Check the documentation for an overview
2. Custom loss functions should be converted into tensorflow graph-compatible functions, which is as easy as putting the tf.function decorator in front of it. See here
3. Loops usually don't do well. Vectorise your functions as much as possible.

All together I reckon something like this should work (didn't test it):

import tensorflow as tf
tf_2pi = tf.constant(tf.sqrt(2*np.pi), dtype=tf.float32)

@tf.function
def kernel(x, sigma=1):
    return (1 / (tf_2pi * sigma)) * tf.exp((-(x * x) / (2 * sigma * sigma)))


@tf.function
def correntropy(y_true, y_pred):
    return -tf.math.reduce_mean(kernel(y_true - y_pred))

Answered By - ThFriedrich