[FIXED] How to fix cardinality error in my CNN model

Issue

I am currently working on a CNN model where my inputs are CSV's with 1080 rows and 4 attributes (columns). I have all the CSVs under their own categories directory, for example /Category A/..., /Category B/... etc.

At start I have created two arrays:

X = []
Y = []

and then in a for loop going thru all the directories I read the contents of the CSV in a 1080x4 shape and put it in my X array like this:

(I have confirmed the values are correctly read and in correct shape)

X.append(pandas.read_csv(itemPath).values)

and I add the category of the item in my Y array right after, so the order of X items are aligned with order of Y items (categories of X values).

Y.append(cat)

Then here is my model, albeit I got it mostly from a Kaggle example, I just wanted to see if it works at all, but I think the values should not give me errors.

model = Sequential()
model.add(Convolution2D(108,4,4, padding= 'valid', data_format='channels_last',input_shape=(1080, 4, 1),activation= 'relu' ))
model.add(Dropout(0.2))
model.add(Flatten())
model.add(Dense(128, activation= 'relu' ))
model.add(Dense(50, activation= 'relu' ))
model.add(Dense(10, activation= 'softmax' ))

model.compile(loss= 'categorical_crossentropy' , optimizer= 'adam' , metrics=[ 'accuracy' ])

and here is where I train my model:

model.fit(X, Y,
      epochs=20,
      batch_size= 20)
score = model.evaluate(X, Y, batch_size=128)

I get the following error:

ValueError                                Traceback (most recent call last)

Input In [103], in <cell line: 1>()
----> 1 model.fit(X, Y,
      2           epochs=20,
      3           batch_size= 20)
      4 score = model.evaluate(X, Y, batch_size=128)

File d:\github\bitirmecnn\venv\lib\site-packages\keras\utils\traceback_utils.py:67, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     65 except Exception as e:  # pylint: disable=broad-except
     66   filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67   raise e.with_traceback(filtered_tb) from None
     68 finally:
     69   del filtered_tb

File d:\github\bitirmecnn\venv\lib\site-packages\keras\engine\data_adapter.py:1655, in _check_data_cardinality(data)
   1651   msg += "  {} sizes: {}\n".format(
   1652       label, ", ".join(str(i.shape[0])
   1653                        for i in tf.nest.flatten(single_data)))
   1654 msg += "Make sure all arrays contain the same number of samples."
-> 1655 raise ValueError(msg)

ValueError: Data cardinality is ambiguous:
  x sizes: 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080
  y sizes: 510
Make sure all arrays contain the same number of samples.

Here is the model summary:

Model: "sequential_12"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d_14 (Conv2D)          (None, 270, 1, 108)       1836      
                                                                 
 dropout_6 (Dropout)         (None, 270, 1, 108)       0         
                                                                 
 flatten_6 (Flatten)         (None, 29160)             0         
                                                                 
 dense_18 (Dense)            (None, 128)               3732608   
                                                                 
 dense_19 (Dense)            (None, 50)                6450      
                                                                 
 dense_20 (Dense)            (None, 10)                510       
                                                                 
=================================================================
Total params: 3,741,404
Trainable params: 3,741,404
Non-trainable params: 0
_________________________________________________________________

Where am I going wrong? The X clearly has 510 entries with dimensions of 1080x4, what should I do to fix this?

Also do you guys have any advice on the model values? I am pretty new into this, I have 510 items and 6 categories

Solution

I used the following code to generate random data and was able to replicate the issue with it.

# Generating random X and y values
x=tf.random.uniform((1080,4,1))    
X=[]
for i in range(510):
  X.append(x)    
y=tf.random.uniform((510,))  

Reshape X to overcome the error as follows:

# Generating random X and y values
x=tf.random.uniform((1080,4,1))
X=[]
for i in range(510):
  X.append(x)

# Reshaping X to overcome the error
X=tf.reshape(X,(510,1080,4,1))
y=tf.random.uniform((510,))  

The units in the output layer must be 6, as your data has 6 categories. Replace the following line

model.add(Dense(10, activation= 'softmax' )) 

with this:

model.add(Dense(6, activation= 'softmax' )) 

Answered By - Tfer3