[FIXED] Langchain pandas agent unable to run pandas commands

Issue

I'm trying to use langchain's pandas agent on python for some development work but it goes into a recursive loop due to it being unable to take action on a thought, the thought being, having to run some pandas code to continue the thought process for the asked prompt on some sales dataset (sales.csv).

here is the below code

import os
os.environ['OPENAI_API_KEY'] = 'sk-xxx'
from langchain.agents import create_pandas_dataframe_agent
from langchain.chat_models import ChatOpenAI
from langchain.llms import OpenAI
import pandas as pd

df = pd.read_csv('sales.csv')
llm = ChatOpenAI(temperature=0.0,model_name='gpt-3.5-turbo')
pd_agent = create_pandas_dataframe_agent(llm, df, verbose=True)
pd_agent.run("what is the mean of the profit?")

and well the response it gives is as below (i replaced ``` with ----)

> Entering new  chain...
Thought: We need to calculate the profit first by subtracting the cogs from the total, and then find the mean of the profit.
Action: Calculate the profit and find the mean using pandas.
Action Input:
----
df['Profit'] = df['Total'] - df['cogs']
df['Profit'].mean()
----
Observation: Calculate the profit and find the mean using pandas. is not a valid tool, try another one.
Thought:I need to use python_repl_ast to execute the code.
Action: Calculate the profit and find the mean using pandas.
Action Input: `python_repl_ast` 
----
df['Profit'] = df['Total'] - df['cogs']
df['Profit'].mean()
----
Observation: Calculate the profit and find the mean using pandas. is not a valid tool, try another one.
Thought:I need to use `python` instead of `python_repl_ast`.
Action: Calculate the profit and find the mean using pandas.
Action Input: `python`
----
import pandas as pd
df = pd.read_csv('filename.csv')
df['Profit'] = df['Total'] - df['cogs']
df['Profit'].mean()
----
.
.
.
.
.
.
Observation: Calculate the profit and find the mean using pandas. is not a valid tool, try another one.
Thought:

> Finished chain.

'Agent stopped due to iteration limit or time limit.'

Now my question is why is it not using the python_repl_ast tool to do the calculation? I even changed this agent's tool's description (python_repl_ast ) which was

A Python shell. Use this to execute python commands. Input should be a valid python command. When using this tool, sometimes output is abbreviated - make sure it does not look abbreviated before using it in your answer.

into

A Python shell. Use this to execute python commands and profit, mean calculation using pandas. Input should be a valid python command. When using this tool, sometimes output is abbreviated - make sure it does not look abbreviated before using it in your answer.

But it did not help. Also i noticed when the python_repl_ast is initialized into my agent the dataframe is loaded into it's local variables tools = [PythonAstREPLTool(locals={"df": df})] so I'm guessing I'm doing something wrong.

Any help will be greatly appreciated. Thank you.

Solution

Most of the information you provide in Langchain agents is supposed to be for prompt context.

Add the below code to provide custom information for your agent.

query = "what is the mean of the profit?"
query = query + " using tool python_repl_ast"
pd_agent.run(query)

It worked for me.

Answered By - Avinash Gaur