[FIXED] loss (I am using mean_absolute_percentage_error) is very high while training for simple regression using tf.keras

Issue

House Prices - Advanced Regression Here is the code:

model = keras.models.Sequential([keras.layers.Flatten(input_shape=[76,1])])
for _ in range(20):
    model.add(keras.layers.BatchNormalization())
    model.add(keras.layers.Activation('selu'))
    model.add(keras.layers.Dense(100, activation='relu'))
model.add(keras.layers.Dense(1))

model.compile(loss='mean_absolute_percentage_error', optimizer=keras.optimizers.SGD(learning_rate=1e-2, decay=2e-4))

early_stopping_cb = keras.callbacks.EarlyStopping(patience=10)
checkpoint_cb = keras.callbacks.ModelCheckpoint("house_prediction_model.h5", save_best_only=True)

history = model.fit(X_train, y_train, epochs=100,
                    validation_data=(X_cv, y_cv),
                    callbacks=[checkpoint_cb, early_stopping_cb])

I thought there is a problem of exploding gradients so I added BatchNormalization. I tried without it that there's no change.

output:

Epoch 1/100
30/30 [==============================] - 1s 24ms/step - loss: 100.0000 - val_loss: 100.0000
Epoch 2/100
30/30 [==============================] - 0s 15ms/step - loss: 99.9999 - val_loss: 100.0000
Epoch 3/100
30/30 [==============================] - 0s 14ms/step - loss: 100.0000 - val_loss: 100.0000
Epoch 4/100
30/30 [==============================] - 1s 19ms/step - loss: 99.9999 - val_loss: 100.0000
Epoch 5/100
30/30 [==============================] - 0s 15ms/step - loss: 99.9999 - val_loss: 100.0000
Epoch 6/100
30/30 [==============================] - 0s 14ms/step - loss: 100.0000 - val_loss: 100.0000
Epoch 7/100
30/30 [==============================] - 0s 12ms/step - loss: 100.0000 - val_loss: 100.0000
Epoch 8/100
30/30 [==============================] - 0s 15ms/step - loss: 99.9999 - val_loss: 100.0000
Epoch 9/100
30/30 [==============================] - 0s 14ms/step - loss: 99.9999 - val_loss: 100.0000
Epoch 10/100
30/30 [==============================] - 0s 15ms/step - loss: 99.9999 - val_loss: 100.0000

Please help me solve this.

Solution

Please checkout these parts in your code:

• input_shape: if your dataset contains 76 features, so you should define it as (76,) and so you don't need to define flatten layer, and change it into input layer with the input shape as i said above.
• layers: it would be better to define your model with low layers (simple architecture) and then if it were not helpful change it into big model. so it would be better to change number of iterations.
• selu: I don't know why did you use selu activation in your model, it's based on an approach or not?
• output_layer: cause or range of values, if the outputs are in high range, it would be better to define lambda layer after the last layer you've defined above, and scale up values based on outputs range something like this (keras.layers.Lambda(lambda val : 100.0 * var))
• optimizer: it would be better optimize the learning rate with a callback (Learning Rate Schedule) if you wanna use SGD as optimizer and based on that define lr. but if you don't wanna optimize learning rate, substitute with adam or rmsprop.
• loss: cause of the vital role the loss function plays in neural networks, you should define function which changes continuously: mean squared error or mean absolute error and huber loss, for this type of problem i think it would be better to change it into huber loss.

Answered By - Soroush Mirzaei