[FIXED] Alternative to .concat() of empty dataframe, now that it is being deprecated?

Issue

I have two dataframes that can both be empty, and I want to concat them.

Before I could just do :

output_df= pd.concat([df1, df2])

But now I run into

FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

An easy fix would be:

if not df1.empty and not df2.empty:
    result_df = pd.concat([df1, df2], axis=0)
elif not df1.empty:
    result_df = df1.copy()
elif not df2.empty:
    result_df = df2.copy()
else:
    result_df = pd.DataFrame()

But that seems pretty ugly. Does anyone have a better solution ?

FYI: this appeared after pandas released v2.1.0

Solution

To be precise, concat is not deprecated (and won't be IMHO) but I can trigger this FutureWarning in 2.1.1 with the following example, while df2 being an empty DataFrame with a different dtypes than df1 :

df1 = pd.DataFrame({"A": [.1, .2, .3]})
df2 = pd.DataFrame(columns=["A"], dtype="object")

out = pd.concat([df1, df2]) ; print(out)

     A
0  0.1
1  0.2
2  0.3

As a solution in your case, you can try something like you did :

out = (df1.copy() if df2.empty else df2.copy() if df1.empty
       else pd.concat([df1, df2]) # if both DataFrames non empty
      )

Or maybe even this one? :

out = pd.concat([df1.astype(df2.dtypes), df2.astype(df1.dtypes)])

Answered By - Timeless