[FIXED] KerasTuner: Custom Metrics (e.g., F1 Score, AUC) in Objective with RandomSearch Error

Issue

I'm using KerasTuner for hyperparameter tuning of a Keras neural network. I would like to use common metrics such as F1 score, AUC, and ROC as part of the tuning objective. However, when I specify these metrics in the kt.Objective during RandomSearch, I encounter issues with KerasTuner not finding these metrics in the logs during training.

Here is an example of how I define my objective:

tuner = kt.RandomSearch(
    MyHyperModel(),
    objective=kt.Objective("val_f1", direction="max"),
    max_trials=100,
    overwrite=True,
    directory="my_dir",
    project_name="tune_hypermodel",
)

But I get:

RuntimeError: Number of consecutive failures exceeded the limit of 3.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/keras_tuner/src/engine/base_tuner.py", line 273, in _try_run_and_update_trial
    self._run_and_update_trial(trial, *fit_args, **fit_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/keras_tuner/src/engine/base_tuner.py", line 264, in _run_and_update_trial
    tuner_utils.convert_to_metrics_dict(
  File "/usr/local/lib/python3.10/dist-packages/keras_tuner/src/engine/tuner_utils.py", line 132, in convert_to_metrics_dict
    [convert_to_metrics_dict(elem, objective) for elem in results]
  File "/usr/local/lib/python3.10/dist-packages/keras_tuner/src/engine/tuner_utils.py", line 132, in <listcomp>
    [convert_to_metrics_dict(elem, objective) for elem in results]
  File "/usr/local/lib/python3.10/dist-packages/keras_tuner/src/engine/tuner_utils.py", line 145, in convert_to_metrics_dict
    best_value, _ = _get_best_value_and_best_epoch_from_history(
  File "/usr/local/lib/python3.10/dist-packages/keras_tuner/src/engine/tuner_utils.py", line 116, in _get_best_value_and_best_epoch_from_history
    objective_value = objective.get_value(metrics)
  File "/usr/local/lib/python3.10/dist-packages/keras_tuner/src/engine/objective.py", line 59, in get_value
    return logs[self.name]
KeyError: 'val_f1'

I would be very thankful if someone could directly guide me to the actual metrics available on the Keras documentation because I have searched and searched, and I can't seem to find them. The only snippet of code that has worked for me is using the accuracy metric like this

import keras_tuner as kt
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.regularizers import l2
from kerastuner.tuners import RandomSearch


class MyHyperModel(kt.HyperModel):
    def build(self, hp):
        model = Sequential()
        model.add(layers.Flatten())
        model.add(
            layers.Dense(
                units=hp.Int("units", min_value=24, max_value=128, step=10),
                activation="relu",
            )
        )
        model.add(layers.Dense(1, activation="sigmoid"))  
        model.compile(
            optimizer=Adam(learning_rate=hp.Float('learning_rate', 5e-5, 5e-1, step=0.001)),#,Adam(learning_rate=hp.Float('learning_rate', 5e-5, 5e-1, sampling='log')),
            loss='binary_crossentropy',
            metrics=['accuracy']
        )
        return model

    def fit(self, hp, model, *args, **kwargs):
        return model.fit(
            *args,
            batch_size=hp.Choice("batch_size", [16, 32,52]),
            epochs=hp.Int('epochs', min_value=5, max_value=25, step=5),
            **kwargs,
        )


tuner = kt.RandomSearch(
    MyHyperModel(),
    objective="val_accuracy",
    max_trials=100,
    overwrite=True,
    directory="my_dir",
    project_name="tune_hypermodel",
)

tuner.search(X_train, y_train, validation_data=(X_test, y_test), callbacks=[keras.callbacks.EarlyStopping('val_loss', patience=3)])

Is it possible that Keras only supports accuracy as the default metric, and we'll have to define any other metric ourselves? I would be very thankful if you could help me find the documentation or kindly show me how to define objective metrics for AUC and F1. Thank you so much!

Solution

The guide Getting started with KerasTuner from the official documentation answers your question:

In all previous examples, we all just used validation accuracy ("val_accuracy") as the tuning objective to select the best model. Actually, you can use any metric as the objective. The most commonly used metric is "val_loss", which is the validation loss.

There are two type of metrics you can use:

• Built-in metrics. You can find the list of built in metrics on the official documentation: Keras Metrics
• Custom Metrics: those metrics, you need to define them yourself by subclassing the Metrics class.

In both case, when defining your Objective, you need to use the name attribute of the Metric object you want to use as an objective. The guide states it quite clearly:

Identify the objective name string. The name string of the objective is always in the format of f"val_{metric_name_string}". For example, the objective name string of mean squared error evaluated on the validation data should be "val_mean_absolute_error".

Note that the metric used as an objective should be compiled with the model:

Compile the model with the the built-in metric. For example, you want to use MeanAbsoluteError(). You need to compile the model with metrics=[MeanAbsoluteError()]. You may also use its name string instead: metrics=["mean_absolute_error"]. The name string of the metric is always the snake case of the class name.

---

So in your case, given that you would like to use a F1 metric as an objective, you need to:

• Compile your model MyHyperModel with the metric. You can use the one defined by TensorFlow if you are using TensorFlow as a backend (or using Keras 2.15), or alternatively, define the metric yourself (See the guide: Creating custom metrics)
• Use the right name when defining your objective. If you use the one defined in TensorFlow, you would use val_f1_score as a name (if you wish to use the f1 score computed on the validation data as your objective).

Repeat those steps with every metric you wish to use.

Answered By - Lescurel