[FIXED] Optuna LightGBM LightGBMPruningCallback

Issue

I am getting an error on my modeling of lightgbm searching for optimal auc. Any help would be appreciated.

import optuna  
from sklearn.model_selection import StratifiedKFold
from optuna.integration import LightGBMPruningCallback
def objective(trial, X, y):
    param = {
        "objective": "binary",
        "metric": "auc",
        "verbosity": -1,
        "boosting_type": "gbdt",
        "lambda_l1": trial.suggest_loguniform("lambda_l1", 1e-8, 10.0),
        "lambda_l2": trial.suggest_loguniform("lambda_l2", 1e-8, 10.0),
        "num_leaves": trial.suggest_int("num_leaves", 2, 256),
        "feature_fraction": trial.suggest_uniform("feature_fraction", 0.4, 1.0),
        "bagging_fraction": trial.suggest_uniform("bagging_fraction", 0.4, 1.0),
        "bagging_freq": trial.suggest_int("bagging_freq", 1, 7),
        "min_child_samples": trial.suggest_int("min_child_samples", 5, 100),
    }


    cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=1121218)

    cv_scores = np.empty(5)
    for idx, (train_idx, test_idx) in enumerate(cv.split(X, y)):
        X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]
        y_train, y_test = y[train_idx], y[test_idx]

        pruning_callback = optuna.integration.LightGBMPruningCallback(trial, "auc")
        
        model = lgb.LGBMClassifier(**param)
        
        model.fit(
            X_train,
            y_train,
            eval_set=[(X_test, y_test)],
            early_stopping_rounds=100,
            callbacks=[pruning_callback])
        
        preds = model.predict_proba(X_test)
        cv_scores[idx] = log_loss(y_test, preds)
        auc_scores[idx] = roc_auc_score(y_test, preds)
        
    return np.mean(cv_scores), np.mean(auc_scores)
    


study = optuna.create_study(direction="minimize", study_name="LGBM Classifier")
func = lambda trial: objective(trial, sample_df[cols_to_keep], sample_df[target])

study.optimize(func, n_trials=1)

Trial 0 failed because of the following error: ValueError('The intermediate values are inconsistent with the objective values in terms of study directions. Please specify a metric to be minimized for LightGBMPruningCallback.',)*

Solution

Your objective function returns two values but you specify only one direction when creating the study. Try this:

study = optuna.create_study(directions=["minimize", "maximize"], study_name="LGBM Classifier")

Answered By - Mutlu Simsek