[FIXED] Is there any equivalent of hyperopts lognormal in Optuna?

Issue

I am trying to use Optuna for hyperparameter tuning of my model.

I am stuck in a place where I want to define a search space having lognormal/normal distribution. It is possible in hyperopt using hp.lognormal. Is it possible to define such a space using a combination of the existing suggest_ api of Optuna?

Solution

You could perhaps make use of inverse transforms from suggest_float(..., 0, 1) (i.e. U(0, 1)) since Optuna currently doesn't provide suggest_ variants for those two distributions directly. This example might be a starting point https://gist.github.com/hvy/4ef02ee2945fe50718c71953e1d6381d Please find the code below

import matplotlib.pyplot as plt
import numpy as np
from scipy.stats import norm
from scipy.special import erfcinv

import optuna


def objective(trial):
    # Suggest from U(0, 1) with Optuna.
    x = trial.suggest_float("x", 0, 1)

    # Inverse transform into normal.
    y0 = norm.ppf(x, loc=0, scale=1)

    # Inverse transform into lognormal.
    y1 = np.exp(-np.sqrt(2) * erfcinv(2 * x))

    return y0, y1


if __name__ == "__main__":
    n_objectives = 2  # Normal and lognormal.

    study = optuna.create_study(
        sampler=optuna.samplers.RandomSampler(),
        # Could be "maximize". Does not matter for this demonstration.
        directions=["minimize"] * n_objectives,
    )
    study.optimize(objective, n_trials=10000)

    fig, axs = plt.subplots(n_objectives)
    for i in range(n_objectives):
        axs[i].hist(list(t.values[i] for t in study.trials), bins=100)
    plt.show()

Answered By - hvy