[FIXED] Neural networks - unable to understand the behaviour of the output layer

Issue

I'd like to know why this works (look for the comment "# This is the output layer and this is what I am talking about"):

model = Sequential() # Not talking about this
model.add(Dense(32, activation='relu', input_dim = X_train.shape[1])) # Not talking about this
model.add(Dense(16, activation='relu')) # Not talking about this
model.add(Dropout(0.2)) # Not talking about this
model.add(Dense(16, activation='relu')) # Not talking about this
model.add(Dense(y_train.nunique()+1, activation='softmax')) # This is the output layer and this is what I am talking about
          
model.compile(loss='sparse_categorical_crossentropy', optimizer=Adam(learning_rate = 0.01), metrics=['accuracy']) # Not talking about this
model.summary() # Not talking about this

and not this (look for the comment "# This is the output layer and this is what I am talking about")::

model = Sequential() # Same as above
model.add(Dense(32, activation='relu', input_dim = X_train.shape[1])) # Same as above
model.add(Dense(16, activation='relu')) # Same as above
model.add(Dropout(0.2)) # Same as above
model.add(Dense(16, activation='relu')) # Same as above
model.add(Dense(y_train.nunique(), activation='softmax')) # This is the output layer and this is what I am talking about
          
model.compile(loss='sparse_categorical_crossentropy', optimizer=Adam(learning_rate = 0.01), metrics=['accuracy']) # Same as above
model.summary() # Same as above

So what's going on here is that I have this very basic NN that I am using to predict a multiclass dataset. There are 10 classes in the target, starting from 0 and going all the way to 10 (with the exception of 9; 9 isn't there). At where I've commented "# This is the output layer and this is what I am talking about", when I give the unit of output neurons as the number of unique values of target (y_train.nunique()), it throws an error like this:

Detected at node sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits defined at (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
...

  File "c:\<redacted>\Projects\<redacted>\Lib\site-packages\keras\src\backend.py", line 5775, in sparse_categorical_crossentropy

Received a label value of 10 which is outside the valid range of [0, 10).  Label values: 5 0 3 3 1 8 10 4 3 1 0 0 1 3 5 6 10 6 10 8 4 6 6 6 1 2 7 10 8 0 4 8
     [[{{node sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits}}]] [Op:__inference_train_function_737445]

On the other hand, when I give the number of units as anything more than that, like in the above example it is y_train.nunique()+1, it goes through all 200 epochs. I don't understand what's going on.

Also,

1. Is the output layer correct (for the number of classes in question)?
2. Is my understanding of the output layer correct (which is - for this specific problem - the number of neurons in the output must be equal to the number of unique values of the target (which are also the classes the data falls into))?

Solution

Yes, I understand your question. The output layer should be equal to the number of target classes. For Example in your case take output neuron as y_train.nunique() is 10 then the final layer output and label should be in the range of 0 - 9. The Label value 10 will be not included in that specific range. So It throw error Received a label value of 10 which is outside the valid range becasuse the label 10 will not include in the final softmax neuron number of 10

The Above image will represent the softmax layer with the classes of 3. [0.9% is class 0, 0.1% is class 1, 0.0% is class 2] If Number of class 10 Only 10 class probability in softmax will be created. In your case the number of classes will be 11.

If y_train.nunique() + 1 is 10 then the final layer output and label should be in the range of 0 - 10 here for the 10th class the probability softmax will be created. So the code works fine.

Sum of all classes probability will be result in 1. The missing 9 is also will be consider as one class

Answered By - MUKILAN S