[FIXED] Why can't bs4 find the table in this website?

Issue

Why can't bs4 find the table in this website? https://www.motogp.com/en/gp-results/2022/JPN/MotoGP/RAC/Classification

Solution

The data is loaded with javascript and so the table is created after your request to the server.

You can either send a request to /classifications (endpoint, that loads the data shown in the table)

r = requests.get("https://www.motogp.com/api/results-front/be/results-api/session/f47a9770-4a91-478b-bb1a-a81899b694ad/classifications")

(note: your session id probably changed)

Or you'd have to use a crawler that can render javascript (like selenium) example:

from selenium import webdriver
driver = webdriver.Chrome()

driver.get("https://www.motogp.com/en/gp-results/2022/JPN/MotoGP/RAC/Classification")
table = driver.find_element(By.TAG_NAME, "table")

Answered By - bitflip