Issue
I am trying to scrape the site https://www.pro-football-reference.com/years/2021/rushing.htm#, and I am trying to sort by the column Y/A
.
Code
from urllib.request import urlopen
from bs4 import BeautifulSoup
url = f'https://www.pro-football-reference.com/years/2021/rushing.htm'
html = urlopen(url)
stats_page = BeautifulSoup(html, features="lxml")
rows = [[i.getText() for i in row.findAll('td')] for row in stats_page.findAll('tr')[1:]]
for row in rows:
if len(row) == 0: rows.remove(row)
rows.sort(key=lambda x: float(x[11]), reverse=True)
Output
[['C.J. Moore', 'DET', '26', '', '8', '0', '1', '28', '0', '1', '28', '28.0', '3.5', '0'], ['Diontae Johnson', 'PIT', '25', 'wr', '6', '6', '1', '25', '0', '1', '25', '25.0', '4.2', '0'], ['Marcus Mariota', 'LVR', '28', '', '2', '0', '2', '35', '0', '1', '31', '17.5', '17.5', '0']...
Problem
While this does technically return what I asked for, I wanted something different. I realized that when I go to the website I see this toggle:
There is a toggle that says "When table is sorted, hide non-qualifiers". When that is toggled, the output should look something more like [[Lamar Jackson, BAL, 24, qb, 7, 7, 76, 480, 2, 29, 31, 6.3, 68.6, 5], [Jalen Hurts, PHI, 23, qb 8, 8, 73, 432, 5, 29, 27, 5.9, 54.0, 5]...]
. But when it's not toggled, it looks like the output I posted above.
However, when you scrape the website it defaults to off. Is there a way to toggle this to be on?
Solution
You could target the table by id, then exclude rows where there are tds
having a class of non_qual
. I would use the html from these rows, wrapped with table
tags, to reconstitute the table with pandas. Finally, sort and tidy the table.
Given there appear to be some ties, within Y/A
, it looks like there is a secondary sort on Att
desc e.g. page output the following order for Y/A 4.5 (current 2021-11-07)
18 Aaron Jones
19 Dalvin Cook
20 Melvin Gordon
21 David Montgomery
Code:
import pandas as pd
from bs4 import BeautifulSoup as bs
import requests
r = requests.get(
'https://www.pro-football-reference.com/years/2021/rushing.htm')
soup = bs(r.content, 'lxml')
t = pd.read_html('<table>' + ''.join([str(r) for r in soup.select(
'#rushing tr:not(:has(td.non_qual))')]) + '</table>')[0]
t.columns = [i[1] for i in t.columns]
t = t[t.Rk != 'Rk'].apply(pd.to_numeric, errors="ignore")
t.sort_values(['Y/A', 'Att'], ascending=[False, False], inplace=True)
t.Rk = [i+1 for i in range(len(t.index))]
t.reset_index(drop=True, inplace=True)
t
Sample output:
Answered By - QHarr
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.