[FIXED] TypeError: '<' not supported between instances of 'str' and 'bool' although info doesn't have bool in sklearn column transformer

Issue

There are similar questions asked before on stackoverflow, however, none of them could fix my problem. I don't understand why info() clearly doesn't output a "bool" but sklearn is outputting an error saying I have boolean values in my dataframe. Can anyone help me debug this thanks!

X = df.drop("Transported", axis=1)
y = df.Transported
X.info()
"""
output:
>>> <class 'pandas.core.frame.DataFrame'>
RangeIndex: 8693 entries, 0 to 8692
Data columns (total 11 columns):
 #   Column        Non-Null Count  Dtype  
---  ------        --------------  -----  
 0   HomePlanet    8492 non-null   object 
 1   CryoSleep     8476 non-null   object 
 2   Cabin         8494 non-null   object 
 3   Destination   8511 non-null   object 
 4   Age           8514 non-null   float64
 5   VIP           8490 non-null   object 
 6   RoomService   8512 non-null   float64
 7   FoodCourt     8510 non-null   float64
 8   ShoppingMall  8485 non-null   float64
 9   Spa           8510 non-null   float64
 10  VRDeck        8505 non-null   float64
dtypes: float64(6), object(5)
memory usage: 747.2+ KB
"""
categorical_features = ["HomePlanet", "CryoSleep", "Cabin", "Destination", "VIP"]
categorical_transformer = Pipeline(steps=[
    ("imputer", SimpleImputer(strategy="constant", fill_value="missing")),
    ("encoder", OneHotEncoder(handle_unknown="ignore"))
])
numerical_features = ["Age", "RoomService", "FoodCourt", "ShoppingMall", "Spa", "VRDeck"]
numerical_transformer = Pipeline(steps=[
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler())
])
preprocessor = ColumnTransformer(transformers=[
    ("cat", categorical_transformer, categorical_features),
    ("num", numerical_transformer, numerical_features)
])
model = Pipeline(steps=[("preprocessor", preprocessor), ("model", RandomForestRegressor())])
X = df.drop("Transported", axis=1)
y = df["Transported"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
model.fit(X_train, y_train)
model.score(X_test, y_test)

error message:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File D:\Programming\python\machine_learning\ml\lib\site-packages\sklearn\utils\_encode.py:173, in _unique_python(values, return_inverse, return_counts)
    171 uniques_set, missing_values = _extract_missing(uniques_set)
--> 173 uniques = sorted(uniques_set)
    174 uniques.extend(missing_values.to_list())

TypeError: '<' not supported between instances of 'str' and 'bool'

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
Input In [68], in <cell line: 19>()
     17 y = df["Transported"]
     18 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
---> 19 model.fit(X_train, y_train)
...

Solution

Thing is, columns like CryoSleep and VIP are actually boolean (I assume this is the original Kaggle ST dataset). They're shown as object because of missing values (resulting in a mixed type).

Try explicitly changing the values first, e.g.:

   df['CryoSleep'] = str(df['CryoSleep'])
   df['VIP'] = str(df['VIP'])

On a minor note, you probably meant using RandomForestClassifier().

Answered By - dx2-66