[FIXED] What happens when I add/remove parameters dynamically during an Optuna study?

Issue

Optuna's FAQ has a clear answer when it comes to dynamically adjusting the range of parameter during a study: it poses no problem since each sampler is defined individually.

But what about adding and/or removing parameters? Is Optuna able to handle such adjustments?

One thing I noticed when doing this is that in the results dataframe these parameters get nan entries for other trials. Would there be any benefit to being able to set these nans to their (default) value that they had when not being sampled? Is the study still sound with all these unknown values?

Solution

Question was answered here:

Thanks for the question. Optuna internally supports two types of sampling: optuna.samplers.BaseSampler.sample_independent and optuna.samplers.BaseSampler.sample_relative.

The former optuna.samplers.BaseSampler.sample_independent is a method that samples independently on each parameter, and is not affected by the addition or removal of parameters. The added parameters are taken into account from the timing when they are added.

The latter optuna.samplers.BaseSampler.sample_relative is a method that samples by considering the correlation of parameters and is affected by the addition or removal of parameters. Optuna's default search space for correlation is the product set of the domains of the parameters that exist from the beginning of the hyperparameter tuning to the present. Developers who implement samplers can implement their own search space calculation method optuna.samplers.BaseSampler.infer_relative_search_space. This may allow correlations to be considered for hyperparameters that have been added or removed, but this depends on the sampling algorithm, so there is no API for normal users to modify.

Answered By - jorijnsmit