[FIXED] Pandas Dataframe: Changes in loop don't show in original

Issue

I am looking back at some old python code that I believe used to work with Pandas v1.x but now fails with Pandas 2 and am stumped.

There are two "input" DataFrames df and df2 that I am trying to loop through and ultimately copy to dfo1 and dfo2, the "output" DataFrames. The use of zip() is complicating (for me) the attempt to define variables as global vs. local.

Question: What is the proper way inside the for loop when using zip() to copy the dataframes so dfo1 and dfo2 can be used after the loop completes?

data=np.random.randn(24).reshape(6,4)
df=pd.DataFrame(data,index=['1','2','3','4','5','6'],columns=['A','B','C','D'])

data2=np.random.randn(24).reshape(6,4)
df2=pd.DataFrame(data,index=['1','2','3','4','5','6'],columns=['A','B','C','D'])

dflist=[df,df2] # Dataframes that serve as Input

print(f'Address of df is:{hex(id(df))}, and df2 is:{hex(id(df2))}')

#Empty output dataframes
dfo1=pd.DataFrame()
dfo2=pd.DataFrame()
dfout=[dfo1,dfo2]

print(f'Address of dfo1 is:{hex(id(dfo1))}, and dfo2 is:{hex(id(dfo2))}')


print('***Begin Loop')
for i,(dfo,dfx) in enumerate(zip(dfout,dflist)):
    print(f'\ti={i}')
    print(f'\t**Start of loop:  Address of dfo is:{hex(id(dfo))}. Address of dfx is:{hex(id(dfx))} ')
    dfx.iloc[3,3]=0 #debug test making change
    
    #Copy doesn't work because of local vs global vars
    #dfo=dfx.copy()

    # ########### The .append() works in v1.x of Pandas. ###########
    #dfo.append(dfx)

    # For v2 of Pandas use concat()?  Dec 2023.  Doesn't change original dfo though!
    dfo=pd.concat([dfo,dfx]) 

    dfo.iloc[0,0]=99 #debug random change for testing
    #End of loop address is different...ie: made a new copy vs. modify original
    print(f'\t**End of loop:  Address of dfo is:{hex(id(dfo))}. Address of dfx is:{hex(id(dfx))} ')
    print(f'\tdfx shape is{dfx.shape}')
    print(f'\tdfo shape is{dfo.shape}')
    #dfout[i]=dfx.copy(deep=True)
    print(dfo)

#TODO:  Why is dfo1 empty after the loop?  Shouldn't be, or that's my intuition anyway
print(f'Address of dfo1 is: {hex(id(dfo1))}')
print(f'dfo1 shape after loop is{dfo1.shape}') #Empty
dfo1   

I tried the above code and various versions of copy(), concat(), etc and nothing works. Can you please point me in the right direction?

Thank You

Solution

My understanding (without trying) is that you cannot affect your values using dfo, because it is indeed used as a variable for the loop. What happens if you replace dfo by dfout[i], at least when writing? You can keep dfo for reading.

Answered By - Gwalchaved