[FIXED] Why does the Optuna CSV file only display 1 item per parameter when I have multiple?

Issue

I have created the following piece of code in Python in order to optimize my network using Optuna.

    activations_1 = trial.suggest_categorical('activation', ['relu', 'sigmoid', 'tanh', 'selu']) 
    activations_2 = trial.suggest_categorical('activation', ['relu', 'sigmoid', 'tanh', 'selu']) 
    activations_3 = trial.suggest_categorical('activation', ['relu', 'sigmoid', 'tanh', 'selu']) 
    activations_4 = trial.suggest_categorical('activation', ['relu', 'sigmoid', 'tanh', 'selu'])

model = Sequential([
            layers.Conv2D(filters=dict_params['num_filters_1'],
                          kernel_size=dict_params['kernel_size_1'],
                          activation=dict_params['activations_1'],
                          strides=dict_params['stride_num_1'],
                          input_shape=self.input_shape),
            layers.BatchNormalization(),
            layers.MaxPooling2D(2, 2),

            layers.Conv2D(filters=dict_params['num_filters_2'],
                          kernel_size=dict_params['kernel_size_2'],
                          activation=dict_params['activations_2'],
                          strides=dict_params['stride_num_2']),

As you can see, I made multiple activation trials instead of one because I wanted to see if the model produced better results when each layer had a different activation function. I did the same with other parameters as you can see. My confusion begins when I return the study.bestparams object:

{"num_filters": 32, "kernel_size": 4, "strides": 1, "activation": "selu", "num_dense_nodes": 64, "batch_size": 64}

The best parameters from the trials produced only one parameter. It does not tell me where the parameter was used and also doesn't display the other 3 activation functions I used (or the other parameters for that matter). Is there a way to precisely display the best settings my model used and at which layers? (I am aware of saving the best model and model summary but this does not help me too much)

Solution

The problem is you used the same parameter name for all activations. Instead of :

    activations_1 = trial.suggest_categorical('activation', ['relu', 'sigmoid', 'tanh', 'selu']) 
    activations_2 = trial.suggest_categorical('activation', ['relu', 'sigmoid', 'tanh', 'selu']) 
    activations_3 = trial.suggest_categorical('activation', ['relu', 'sigmoid', 'tanh', 'selu']) 
    activations_4 = trial.suggest_categorical('activation', ['relu', 'sigmoid', 'tanh', 'selu'])

Try:

    activations_1 = trial.suggest_categorical('activation1', ['relu', 'sigmoid', 'tanh', 'selu']) 
    activations_2 = trial.suggest_categorical('activation2', ['relu', 'sigmoid', 'tanh', 'selu']) 
    activations_3 = trial.suggest_categorical('activation3', ['relu', 'sigmoid', 'tanh', 'selu']) 
    activations_4 = trial.suggest_categorical('activation4', ['relu', 'sigmoid', 'tanh', 'selu'])

Answered By - dankal444