Issue
The table on this page needs to be scraped daily. We are trying to keep the scraping as simple (robust) as possible so there are no issues with the code running on our server. Would like to steer clear of Selenium:
import requests
import pandas as pd
page_list = pd.read_html('https://www.ncaa.com/rankings/basketball-women/d1/ncaa-womens-basketball-net-rankings')
page_df = pd.DataFrame(page_list)
# won't convert to df (ValueError: Must pass 2-d input. shape=(1, 356, 9)
r = requests.get('https://www.ncaa.com/rankings/basketball-women/d1/ncaa-womens-basketball-net-rankings')
# not sure what to do with response
page_list is close but it is a 3-dimensional list. How can we get this into a 2-dimensional list, or into a pandas dataframe?
Solution
No need to do page_df = pd.DataFrame(page_list[0]). Can actually simply this to page_df = page_list[0]:
page_list = pd.read_html('https://www.ncaa.com/rankings/basketball-women/d1/ncaa-womens-basketball-net-rankings')
page_df = page_list[0]
Answered By - chitown88
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.