[FIXED] Web Scraping NBA Combine data

Issue

I am trying to web scrape the table from this webpage. However, a table is not being generated from what I have on script so far.

import requests
import pandas as pd
from bs4 import BeautifulSoup

url = "https://www.nba.com/stats/draft/combine-anthro?SeasonYear=2021-22"

response = requests.get(url)
response.status_code

response.content

response = requests.get(url).content
soup = BeautifulSoup(response, "html.parser")

soup.find_all("table")

Does anybody have advice on how I can retrieve it? Thanks in advance.

Solution

You can try catching the JSON returned by an external URL :

import requests

url = "https://stats.nba.com/stats/draftcombineplayeranthro"

payload = {"LeagueID": "00", "SeasonYear": "2022-23"}

headers = {
    "Referer": "https://www.nba.com/",
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64)"
}

data = requests.get(url, params=payload, headers=headers).json()

df = pd.DataFrame(data["resultSets"][0]["rowSet"],
                  columns=data["resultSets"][0]["headers"])

Output :

print(df)

 TEMP_PLAYER_ID  PLAYER_ID FIRST_NAME  ... BODY_FAT_PCT HAND_LENGTH HAND_WIDTH
        1630534    1630534      Ochai  ...         5.40        8.75       9.50
        1631116    1631116    Patrick  ...         8.90        8.75       9.50
        1631094    1631094      Paolo  ...          NaN         NaN        NaN
            ...        ...        ...  ...          ...         ...        ...
        1631109    1631109       Mark  ...         5.40        9.00       9.75
        1630592    1630592      Jalen  ...          NaN         NaN        NaN
        1630855    1630855      Fanbo  ...          NaN         NaN        NaN

[83 rows x 18 columns]

Answered By - Timeless