[FIXED] Get the S&P 500 tickers list

Issue

So I am using this series on Python for Finance and it keeps giving me error --

1) line 22, in <module> save_sp500_tickers() and 

2) line 8, in save_sp500_tickers
    soup = bs.BeautifulSoup(resp.text,'lxml')and 

3) line 165, in __init__
    % ",".join(features))
bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: lxml.
Do you need to install a parser library?

I have been at it for a whole day and I honestly refuse to give up and any help with this would be greatly appreicated. Also if anyone has any suggestions for something other than pickle and can help write something that allows me to call the SP500 without pickle that would be great.

import bs4 as bs    
import pickle    
import requests    
import lxml    
def save_sp500_tickers():
    resp = requests.get('https://en.wikipedia.org/wiki/List_of_S%26P_500_companies')        
    soup = bs.BeautifulSoup(resp.text,'lxml')        
    table = soup.find('table', {'class': 'wikitable sortable'})        

    tickers = []

    for row in table.findAll('tr')[1:]:
        ticker = row.findAll('td')[0].text
        tickers.append(ticker)

    with open("sp500tickers.pickle", "wb") as f:
        pickle.dump(tickers, f)
    print(tickers)

    return tickers    

save_sp500_tickers()

Solution

Running your code as-is works on my system. Probably, as Eric suggests, you should install lxml.

Unfortunately if you are on Windows pip install lxml does not work unless you have a whole compiler infrastructure set up, which you probably don't.

Luckily you can get a precompiled binary installer from http://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml - make sure you pick the one that matches your version of python and whether it is 32 or 64 bit.

Edit: just for interest, try changing the line

soup = bs.BeautifulSoup(resp.text, 'html.parser')   # use Python's built-in parser instead

See https://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser for a list of available parsers.

Answered By - Hugh Bothwell