[FIXED] How to deal with missing values in Pandas DataFrame?

Issue

I have a Pandas Dataframe that has some missing values. I would like to fill the missing values with something that doesn't influence the statistics that I will do on the data.

As an example, if in Excel you try to average a cell that contains 5 and an empty cell, the average will be 5. I'd like to have the same in Python.

I tried to fill with NaN but if I sum a certain column, for example, the result is NaN. I also tried to fill with None but I get an error because I'm summing different datatypes.

Can somebody help? Thank you in advance.

Solution

there are many answers for your two questions.

Here is a solution for your first one:

If you wish to insert a certain value to your NaN entries in the Dataframe that won't alter your statistics, then I would suggest you to use the mean value of that data for it.

Example:

df # your dataframe with NaN values

df.fillna(df.mean(), inplace=True)

For the second question:

If you need to check descriptive statistics from your dataframe, and that descriptive stats should not be influenced by the NaN values, here are two solutions for it: 1)

df # your dataframe with NaN values

df.fillna(df.mean(), inplace=True)

df.mean()
df.std()

# or even:

df.describe()

2) Option 2:

I would suggest you to use the numpy nan functions such as (numpy.nansum, numpy.nanmean, numpy.nanstd)...

df.apply(numpy.nansum)

df.apply(numpy.nanstd) #...

Answered By - Philipe Riskalla Leal