[FIXED] groupby multiple columns and broadcast results back to each row in dataframe

Issue

I have posted this question before, but it keeps getting closed due to having similar questions asked, but those solutions have not helped me here.

I have a dataframe that needs to be grouped by 3 different columns. From the resultant groupings, I need to perform calculations and then apply the result to each row in a new column.

My data looks like this:

ID   Deal  Party  Commodity       startdate  enddate  fixedpricestrike  quantity  mtmvalue
---- ----- -----  ---------       ---------  -------  ----------------  --------  ---------
J1   Sell   J   (stock1, stock2)   01Jan23   01Feb23        10.00          10      100.00
J2   Sell   J   (stock1, stock2)   01Jan23   01Feb23        10.00          10      100.00
J3   Buy    J   (stock1, stock2)   01Jan23   01Feb23        5.00           10      50.00
J4   Buy    J   (stock1, stock2)   01Jan23   01Feb23        5.00           5       25.00

My objective is to group the data by [Deal, commodity, startdate] so that the resultant data looks like this:

ID   Deal  Party  Commodity       startdate  enddate  fixedpricestrike  quantity  mtmvalue
---- ----- -----  ---------       ---------  -------  ----------------  --------  ---------
J1   Sell   J   (stock1, stock2)   01Jan23   01Feb23        10.00          10      100.00
J2   Sell   J   (stock1, stock2)   01Jan23   01Feb23        10.00          10      100.00

ID   Deal  Party  Commodity       startdate  enddate  fixedpricestrike  quantity  mtmvalue
---- ----- -----  ---------       ---------  -------  ----------------  --------  ---------
J3   Buy    J   (stock1, stock2)   01Jan23   01Feb23        5.00           10      50.00
J4   Buy    J   (stock1, stock2)   01Jan23   01Feb23        5.00           5       25.00

From this, I need to use a formula to calculate a 'fprice' and add it to each row like this:

ID   Deal  Party  Commodity       startdate  enddate  fixedpricestrike  quantity  mtmvalue fprice
---- ----- -----  ---------       ---------  -------  ----------------  --------  --------- -----
J1   Sell   J   (stock1, stock2)   01Jan23   01Feb23        10.00          10      100.00    0
J2   Sell   J   (stock1, stock2)   01Jan23   01Feb23        10.00          10      100.00    0

ID   Deal  Party  Commodity       startdate  enddate  fixedpricestrike  quantity  mtmvalue fprice
---- ----- -----  ---------       ---------  -------  ----------------  --------  --------- -----
J3   Buy    J   (stock1, stock2)   01Jan23   01Feb23        5.00           10      50.00     1.25
J4   Buy    J   (stock1, stock2)   01Jan23   01Feb23        5.00           10      25.00     1.25

My issue lies in the next step, when I try to add the fprice back to the original dataframe I have this line of code:

df['fprice'] = df.groupby(['StartDate', 'Commodity', 'Deal']).apply(lambda group: -(group['MTMValue'].sum() - (group['FixedPriceStrike'] * group['Quantity']).sum()) / group['Quantity'].sum()).reset_index(drop=True)

which returns this dataframe:

ID   Deal  Party  Commodity       startdate  enddate  fixedpricestrike  quantity  mtmvalue fprice
---- ----- -----  ---------       ---------  -------  ----------------  --------  --------- -----
J1   Sell   J   (stock1, stock2)   01Jan23   01Feb23        10.00          10      100.00    0
J2   Sell   J   (stock1, stock2)   01Jan23   01Feb23        10.00          10      100.00    1.25
J3   Buy    J   (stock1, stock2)   01Jan23   01Feb23        5.00           10      50.00     
J4   Buy    J   (stock1, stock2)   01Jan23   01Feb23        5.00           10      25.00     

when the result should look like

ID   Deal  Party  Commodity       startdate  enddate  fixedpricestrike  quantity  mtmvalue fprice
---- ----- -----  ---------       ---------  -------  ----------------  --------  --------- -----
J1   Sell   J   (stock1, stock2)   01Jan23   01Feb23        10.00          10      100.00    0
J2   Sell   J   (stock1, stock2)   01Jan23   01Feb23        10.00          10      100.00    0
J3   Buy    J   (stock1, stock2)   01Jan23   01Feb23        5.00           10      50.00     1.25
J4   Buy    J   (stock1, stock2)   01Jan23   01Feb23        5.00           10      25.00     1.25

I am also relatively new to using pandas, and I am unsure why my result is coming out this way. Any suggestions would help

Solution

Instead of doing it in one step, you can first calculate the 'fprice' and then merge it back to the original DataFrame:

# Calculate 'fprice' for each group
grouped = df.groupby(['startdate', 'Commodity', 'Deal']).apply(
    lambda group: -(group['mtmvalue'].sum() - (group['fixedpricestrike'] * group['quantity']).sum()) / group['quantity'].sum()
).reset_index(name='fprice')

# Merge 'fprice' back to the original DataFrame
df = pd.merge(df, grouped, on=['startdate', 'Commodity', 'Deal'], how='left')

Answered By - Eduardo Motta de Moraes