[FIXED] Custom metric function for TensorFlow doesn't work correctly, it's always outputs zero

Issue

I have y-value of this type: <tf.Tensor: shape=(4,), dtype=float32, numpy=array([ 1., 1.1, 57.9, 1.0243483],type=float32)> custom_binary_crossentropy works fine

def custom_binary_crossentropy(y_true, y_pred): 
    y_true_first = y_true[:, 0]
    y_pred_expanded = tf.expand_dims(y_pred, axis=-1)
    # Calculate binary crossentropy between y_true_first and y_pred
    loss = tf.keras.losses.binary_crossentropy(y_true_first, y_pred_expanded)
    return loss

But custom metric function doesn't:

def up_profit_metric(y_true, y_pred):
    y_pred_class = tf.cast(tf.argmax(y_pred, axis=-1), tf.int32)
    answer = tf.cast(y_true[:, 0], tf.int32)
    go_high = y_true[:, 1]
    price_now = y_true[:, 2]
    perc_inH = y_true[:, 3]
    q_to_buy = tf.cast(tf.round(1000 / price_now), tf.float32)
    deal_amt = q_to_buy * price_now
    # True positive condition
    condition1 = tf.logical_and(tf.equal(answer, y_pred_class), tf.equal(y_pred_class, tf.constant([1])))
    profit1 = deal_amt * go_high / 100
    # False positive condition
    condition2 = tf.logical_and(tf.not_equal(answer, y_pred_class), tf.equal(y_pred_class, tf.constant([1])))
    profit2 = deal_amt * perc_inH / 100
    # Select the appropriate profit based on conditions
    profit = tf.where(condition1, profit1, tf.where(condition2, profit2, tf.cast(0, dtype=tf.float32)))
    # Sum the profits
    total_profit = tf.reduce_sum(profit)
    return total_profit

It does work on made-up example:

y_true_example = tf.constant([[1.0, 1.1, 45.91, 1.26], [1, 0.9, 30.0, 0.1], [0.0, 0.9, 30.0, 0]], dtype=tf.float32)
y_pred_example = tf.constant([[0.2, 0.8], [0.3, 0.7], [0.3, 0.7]], dtype=tf.float32)
profit_example = up_profit_metric(y_true_example, y_pred_example)

But doesn't work in actual model, it outputs zeros on every epoch:

lstm_up2 = Sequential()
lstm_up2.add(LSTM(8, activation='LeakyReLU', dropout = .0))  #linear  relu , activation='relu' LeakyReLU
lstm_up2.add(Dense(1, activation='sigmoid'))
lstm_up2.compile(optimizer='adam', loss=custom_binary_crossentropy, metrics = [up_profit_metric]) 
es_up = EarlyStopping(monitor='val_precision', mode='max', min_delta = 0.01,verbose=1, patience=25)
lstm_up2.fit(
    X_train_reshaped, y_tr_h, y_tr_h, 
    validation_data=(X_val_reshaped, y_val_h), 
    epochs=70, batch_size=24, shuffle=False            )

I've spent 3 days trying solve it, for that I've registered here

Solution

That's because your model output has 1 unit and shape (n,) and your example output has shape (n, 2). So your custom metric always evaluates to the same thing because the argmax operation always returns a 0.

If you're using this model as a classifier, you need n output units to represent n possible categories. Then, possibly a sigmoid or softmax activation depending on the mutual exclusivity of the categories.

This always returns 0:

y_pred_class = tf.cast(tf.argmax(y_pred, axis=-1), tf.int32)

and this later line always evaluates to False:

condition1 = tf.logical_and(
    tf.equal(answer, y_pred_class), 
    tf.equal(y_pred_class, tf.constant([1])) # this never is true
)

Exhibit A - this doesn't work

for i in range(10):
    inputs = tf.random.uniform((3, 1))
    print(up_profit_metric(y_true_example, inputs))

tf.Tensor(0.0, shape=(), dtype=float32)
tf.Tensor(0.0, shape=(), dtype=float32)
tf.Tensor(0.0, shape=(), dtype=float32)
tf.Tensor(0.0, shape=(), dtype=float32)
tf.Tensor(0.0, shape=(), dtype=float32)
tf.Tensor(0.0, shape=(), dtype=float32)
tf.Tensor(0.0, shape=(), dtype=float32)
tf.Tensor(0.0, shape=(), dtype=float32)
tf.Tensor(0.0, shape=(), dtype=float32)
tf.Tensor(0.0, shape=(), dtype=float32)

Exhibit B - this works

for i in range(10):
    inputs = tf.random.uniform((3, 2)) # if you had two output units
    print(up_profit_metric(y_true_example, inputs))

tf.Tensor(8.91, shape=(), dtype=float32)
tf.Tensor(20.020222, shape=(), dtype=float32)
tf.Tensor(11.110221, shape=(), dtype=float32)
tf.Tensor(20.020222, shape=(), dtype=float32)
tf.Tensor(8.91, shape=(), dtype=float32)
tf.Tensor(8.91, shape=(), dtype=float32)
tf.Tensor(8.91, shape=(), dtype=float32)
tf.Tensor(8.91, shape=(), dtype=float32)
tf.Tensor(11.110221, shape=(), dtype=float32)
tf.Tensor(20.020222, shape=(), dtype=float32)

Answered By - Nicolas Gervais