[FIXED] Imputing only the numerical values using sci-kit learn

Issue

So, I have a DataFrame that contains a mix of both Categorical and Numerical values that is currently 12345 rows by 171 columns.

I have missing values in both the Categorical variable and the Numerical where I would like to impute the values. For the Numerical columns I am doing the following;

import pandas as pd
import numpy as np

data = pd.read_csv('filepath')

from sklearn.preprocessing import Imputer
imp = Imputer(missing_values=np.nan, strategy='mean', axis=0)
data = imp.fit_transform(data)

I then get the following error

ValueError: could not convert string to float: 'USD'

This I understand is because I am using sci-kit learns imputer with strategy='mean' which is not compatible with Categorical variables. I would rather not have to go through each column and manually pull out the Numerical values so I am looking for a way that I can perform this imputation only on the Numerical columns.

Solution

If you use panda's categorical encoding functionality, this can all be handled pretty simply. I however rarely find myself with properly encoded data, and would rather have a robust solution than rely on pure pandas.

Here's what I would do.

categorical_columns = []
numeric_columns = []
for c in data.columns:
    if data[c].map(type).eq(str).any(): #check if there are any strings in column
        categorical_columns.append(c)
    else:
        numeric_columns.append(c)

#create two DataFrames, one for each data type
data_numeric = data[numeric_columns]
data_categorical = pd.DataFrame(data[categorical_columns])


from sklearn.preprocessing import Imputer
imp = Imputer(missing_values=np.nan, strategy='mean', axis=0)
data_numeric = pd.DataFrame(imp.fit_transform(data_numeric), columns = data_numeric.columns) #only apply imputer to numeric columns


#you could do something like one-hot-encoding of data_categorical here

#join the two masked dataframes back together
data_joined = pd.concat([data_numeric, data_categorical], axis = 1)

Answered By - skoeb