[FIXED] What would cause the packages/libraries in Spyder to be different from the ones when using pip?

Issue

This problem occurred when I was trying to import pandas: df = pd.read_html('https://www.macrotrends.net/stocks/charts/BMO/Bank-of-Montreal/dividend-yield-history')

I got the error message: raise ImportError("lxml not found, please install it")

ImportError: lxml not found, please install it

So then I pip lxml to install, same error. then I pip3 lxml, same issue...

And this led me down a rabbit hole of installing and re-installing lxml, pandas, html5lib, html and then deleting all of Python from my machine as well as Spyder, then re-installing everything and still having the same issue of that error "lxml not found..."

So then I did 'pip list' on command line, lxml and pandas and everything else I installed is there.

I found some code to find all the packages/libraries installed on Spyder using this: import pkg_resources

dists = [str(d).replace(" ","==") for d in pkg_resources.working_set]
for i in dists:
    print(i)

And I get:

runfile('C:/Users/Preston_Jarvis/untitled0.py', wdir='C:/Users/Preston_Jarvis')
zipp==3.4.1
yapf==0.31.0
wheel==0.36.2
webencodings==0.5.1
wcwidth==0.2.5
And so on...

Pandas is there, but lxml or html5lib or hmtl

What could be causing this?

Thank you

Solution

How did you install Spyder? Did you use the installer or install it with pip/conda? Where are you installing your packages - what environment?

It appears that the environment where you install packages with pip (e.g. lxml) is different than what Spyder is running with. See https://docs.spyder-ide.org/current/faq.html#using-existing-environment

Note that for Spyder to work with other environments you need to install spyder-kernels in this environment (e.g. pip install spyder-kernels)

Answered By - rhkarls