[FIXED] When using the optuna plugin for hydra, can I import the search space from another config file?

Issue

I want to hyper-parameter optimize multiple time series forecasting models on the same data. I'm using the Optuna Sweeper plugin for Hydra. The different models have different hyper-parameters and therefore different search spaces. At the moment my config file looks like this:

defaults:
  - datasets: data
  - models: Ets
  - override hydra/sweeper: optuna
  - override hydra/sweeper/sampler: tpe

hydra:
#  launcher:
#    n_jobs: 10
 run:
  dir: data/outputs/${now:%Y-%m-%d}/${user.user}/${now:%H-%M-%S}
 sweeper:
   sampler:
     seed: 123
   direction: minimize
   study_name: main_val
   storage: null
   n_trials: 2
   n_jobs: 4

   search_space: 

# Ets
    models.damped_trend:
      type: categorical
      choices:
      - 'True'
      - 'False'

  # Theta
    # models.method:
    #   type: categorical
    #   choices:
    #   - 'additive'
    #   - 'multiplicative' 

Now, when I run the main_val.py file with --multirun, I get the optimal hyper-parameters for Ets. Great. But when I want to run the optimization for another model, in this example Theta, I have to manually comment out the search space for Ets and uncomment the search space for Theta. In reality, each model has much more parameters to optimize and I'm working with 10 different models. This makes my config file quite long and confusing and this commenting/uncommenting stuff is both annoying and error-prone.

I would like to import the search space for each model from another yaml file. Is that possible?

I tried the following:

defaults:
  - datasets: data
  - models: Ets
  - search_spaces: Ets
  - override hydra/sweeper: optuna
  - override hydra/sweeper/sampler: tpe

hydra:
#  launcher:
#    n_jobs: 10
 run:
  dir: data/outputs/${now:%Y-%m-%d}/${user.user}/${now:%H-%M-%S}
 sweeper:
   sampler:
     seed: 123
   direction: minimize
   study_name: main_val
   storage: null
   n_trials: 2
   n_jobs: 4

   search_space: search_spaces

with the file search_spaces/Ets.yaml looking like this:

models.damped_trend:
  type: categorical
  choices:
  - 'True'
  - 'False'

But I got the error:

Validation error while composing config:
    Cannot assign str to Dict[str, Any]
        full_key: hydra.sweeper.search_space
        object_type=OptunaSweeperConf

Solution

Here are two options:

1. Use a @package directive
2. Use a variable interpolation

In detail:

Using an @package directive

An @package directive can be used to place Ets.yaml in the hydra.sweeper.search_space package:

defaults:
  - datasets: data
  - models: Ets
  - [email protected]_space: Ets
  - override hydra/sweeper: optuna
  - override hydra/sweeper/sampler: tpe

hydra:
 run:
  dir: data/outputs/${now:%Y-%m-%d}/${user.user}/${now:%H-%M-%S}
 sweeper:
   sampler:
     seed: 123
   direction: minimize
   study_name: main_val
   storage: null
   n_trials: 2
   n_jobs: 4

Using a variable interpolation:

A string interpolation can be used to create a reference from hydra.sweeper.search_spaces to the top-level search_spaces config.

defaults:
  - datasets: data
  - models: Ets
  - search_spaces: Ets
  - override hydra/sweeper: optuna
  - override hydra/sweeper/sampler: tpe

hydra:
 run:
  dir: data/outputs/${now:%Y-%m-%d}/${user.user}/${now:%H-%M-%S}
 sweeper:
   sampler:
     seed: 123
   direction: minimize
   study_name: main_val
   storage: null
   n_trials: 2
   n_jobs: 4

   search_space: ${search_spaces}

Answered By - Jasha